diff --git a/CHANGELOG.md b/CHANGELOG.md index 2f8420f..c4139de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,24 @@ All notable changes to Runloop. The format follows [Keep a Changelog](https://ke ## [Unreleased] ### Added +- **The immersive town (DL-014, 2026-10-18).** `/` is a game you stay in rather than a web page with a canvas. + - **Full screen:** the town is the whole viewport, with no sidebar or top bar; the land goes on past the town (outskirts, farms, fields, the Brook and a lake, forest, hills, mountains) into haze and a sky that follows sim time. + - **Camera:** perspective with a narrow field of view (the default still reads isometric); drag to pan with inertia, zoom toward the cursor, free rotation (Q/E, right-drag, two-finger twist), tilt 10°–62° (Z/X), WASD/arrows, soft bounds, follow, fly into a building with the roof lifting. `cam=x,z,zoom[,az,tilt]` links (3-part links still work); `window.__runloopCam` for tests. + - **HUD:** world chip, Find (⌘K), the bell, the clock and speed with the sound toggle, notifications, a minimap (view outline, agents with status glyphs, alerts; click to go), the hotbar (Explore, Agents, Add to World, Intelligence, Lab, Menu), entity cards anchored next to their entity, the guide banner, an Esc menu (Display, Sound, Controls with the Simulated-vs-Real legend, Runtime, Connections, Pages, Help). It fades while you drag and hides with H; a needs-you dot stays. + - **Everything in place:** Add to World picks a lot in 3D (free lots are outlined and their labels are buttons) and collects the name and workspace in a card on the lot; Intelligence is the World Log at the Commons Library with any run's full trace in a sheet; Lab is compare at the Lattice Lab; Sessions and Connections are sheets; inside an agent's building a session strip scrubs its recorded sessions. `/?overlay=trace|compare|log|sessions|connect|list|map|add` opens them by link. The legacy pages stay full pages, each with "Open in the world". + - **Decorative ambient life:** cars on the roads out, bird flocks, drifting clouds and cloud shadows, water shimmer, a windmill. Never pickable or listed, never within 12 cells of an agent lot, still when the world is paused, gone under reduced motion. + - **Sound** (`apps/studio/src/audio`, Web Audio only, no files or dependencies): an ambient bed that follows the camera and the time of day, interface sounds, and four cues for live, real agent events only (goes live, needs approval, incident, task done), rate-limited; no sound before the first gesture, silent in a hidden tab, volumes and "reduce sound" in Esc › Sound, `window.__runloopAudio` for tests. + - **Phones** get a light 3D town with touch gestures (one-finger pan, pinch, twist, two-finger tilt) and bottom-sheet cards; framings keep room for the card or session strip at the bottom, so entering a building shows its diorama; the 2D map stays for WebGL off and as a Display setting. + - **Navigation:** `useStudioNav()` sends every run link to the right place: sheets and buildings on `/`, the pages elsewhere (⌘K included). + - **Compare** shows Claude Code's reported cost for dispatched runs (a Cost tile; "not reported" otherwise), says when a run is still recording ("Same calls so far…", "As of now…"), words the core's caveats (a run that did not finish, different outcomes, different test scope) and marks a run that no longer finishes as a warning; Duration says "excluding time paused". +- **The town (Runloop Commons), now the Studio's home (`/`).** A persistent, deterministic world beside the run log (DL-009, DL-010; plan `docs/architecture/world-simulation.md`). + - **Simulation** (`packages/town`, `packages/sim`, pure): 80 residents with needs, schedules, traits and friendships deciding by integer scoring; 28 buildings with interiors and opening hours; 14 outdoor spots; one tick per sim minute at 0/1/4/16×. `applyEvent` is the only mutator, shared by the runtime, replay and the Studio; `explain` builds a resident's Why only from its recorded decision. + - **World log** with a provenance on every event (`simulation`, `user`, `external-agent`, `system`, `inferred`), 15 event types, commands checked against `ALLOWED_SOURCES`; `WorldHost` commits before publishing, checkpoints, resumes after a restart (`world.resumed_after_offline`); migrations 3 and 4. + - **API:** `/api/world` (info, town, snapshot, events, explain, SSE stream, commands); `RUNLOOP_WORLD=off` turns the town off. + - **Studio:** a 3D town (merged static scene, instanced residents with a shader rig and level of detail, sky and lit windows from sim time (DL-011), cutaway interiors, labels), the clock and speed controls, a mode rail (Explore, Agents, Intelligence, Lab), context panels (resident with Why, building, place, district, agent), the World Log, an attention ticker, a map and a keyboard directory, ⌘K for residents and buildings, deep links (`?mode&focus&cam`) and redirects from old `/?run=` links. Automatic render quality (the new default) steps down when frames run slow. +- **Real agents in the town** (DL-012). **Add to World** gives a Claude Code agent a building in the Agent District for one project folder; sessions bind by dispatch id, then by folder, else wait in **Arrivals** to be claimed (claiming replays earlier runs). Health (`not_connected`, `awaiting_first_event`, `live`, `idle`, `offline`, `unknown`), attention, tool counts, subagents and incidents come only from the hooks, each citing its run events. The building's interior shows the session's robots. API: `/api/agents`, `/api/agents/arrivals`, `/api/agents/:id/claim`. +- **Assign task (dispatch).** Preview the exact `claude -p` command (preset, model, budget, timeout, environment variable names), approve it with a single-use nonce, follow its output, cancel it. `dontAsk` with an allowlist, haiku by default, a budget Claude Code enforces, its own process group, an audit log. `--plugin-dir` is added unless `claude plugin list` reports the tracer installed. `RUNLOOP_DISPATCH=off` turns it off; `RUNLOOP_DISPATCH_ENV` passes extra variable names. +- **Scripts:** `scripts/e2e-world.mjs` (the town renders, residents move, the clock advances, ⌘K opens a resident's Why with linked factors, Follow, the café interior); `scripts/perf-town.mjs` (render budgets under SwiftShader); `pnpm --filter @runloop-world/sim bench`. `pnpm screenshots` now captures the town, its panels, Add to World, an agent, its interior, Assign task and the World Log (`--suffix` for a second pass, e.g. at night). - **Run compare (Experiments).** Pin a run as a baseline, run the agent again, compare. - **Experiments page** (`/experiments?a=&b=`): the baseline defaults to the newest pinned run and the candidate to the newest other run of the same task. Six summary tiles (outcome, tests passing, failures, duration, tool calls, tokens), the first divergence, outcome and tests (fixed, newly failing, still failing), files only in A, only in B or in both, tool activity by category, tokens and models per agent, and permissions and delegation. Test runs, failures, files, categories, usage reports and the divergence link to their events in Traces. - **Honesty rules:** a value a side did not report shows "not reported" and is never compared or shown as 0; executed-vs-traced pairs, runs still recording and token totals that cover different agents are flagged; a target or test name one side did not report is unknown, never a difference. @@ -12,7 +30,7 @@ All notable changes to Runloop. The format follows [Keep a Changelog](https://ke - **Core:** `summarizeRun` and `diffRuns` in `packages/core/src/compare.ts`, pure and unit-tested. - **`run.started.speed`** (optional, additive): the scripted demo's pacing, so compare does not call a run at speed 0 "faster" than one at speed 1. - **`tool.call_started.target`** (optional, additive): the file of Read/Edit/Write/NotebookEdit, the directory of Grep/Glob, the first word of a shell command (never its arguments); the workspace path or program for the scripted demo's tools. -- **3D world richness pass (D10):** tighter framing (at 1440×900 the board spans 97% of the stage width and a robot projects to 44 px; at 1920×1080, 81–82%), chibi robots with a dark glass visor and state-coloured eyes, warm key and cool rim light, tinted rooms with shaded floors, paths that light up only while a robot walks them or a recorded message crosses them, and a light-theme "architectural model" look (DL-009). +- **3D world richness pass (D10):** tighter framing (at 1440×900 the board spans 97% of the stage width and a robot projects to 44 px; at 1920×1080, 81–82%), chibi robots with a dark glass visor and state-coloured eyes, warm key and cool rim light, tinted rooms with shaded floors, paths that light up only while a robot walks them or a recorded message crosses them, and a light-theme "architectural model" look (DL-013). - **Screenshots:** `scripts/screenshots.mjs` also captures Experiments (default pair, each run kind's pair and its lower half, an unknown run, offline) and a pinned baseline's Traces header. It pins a baseline for the capture only when none exists, and unpins it afterwards (`--no-pin` to skip). - **Claude Code tracer.** The plugin is in `integrations/claude-code-plugin` and the marketplace manifest in `.claude-plugin/marketplace.json`. - Hooks are observe-only: async, with Stop, StopFailure and SessionEnd synchronous. They always exit 0 and print nothing. @@ -48,6 +66,13 @@ All notable changes to Runloop. The format follows [Keep a Changelog](https://ke - **Docs:** `docs/architecture/overview.md`, `docs/architecture/event-model.md` and `docs/status/verification-report.md`. ### Changed +- **Assign task approval moved to the runtime.** The Studio shows the exact command, then **Approve in Runloop…** opens the runtime's own approval page; it follows the outcome (`GET /api/dispatches/previews/:id`) and never posts a nonce itself. +- **Agent truth labels:** "Real · Claude Code · observed" only for an agent registered in this runtime; a presence the agents API does not list is an "Unregistered building" with no actions, and one not confirmed yet says so. A dispatched task whose hooks have not reported yet reads "Task running · no events yet" beside the health; "Live" now means a session is open and its hooks reported in the last 5 minutes. +- **World clock:** clock frames are anchored at the runtime's `wallMs`, and a speed change re-bases the clock at its event, so pausing at 16× no longer throws walkers back and steady 16× no longer steps backwards; resuming the stream names the world (`?after=&world=`), and a snapshot from another world resets the town's state instead of folding onto it. +- `scripts/screenshots.mjs` captures the immersive states (horizon, tilted street, the café, Add to World's lots and card, the agent and its building, the Intelligence and Lab sheets, the trace sheet, the Esc menu); `scripts/e2e-world.mjs` waits for clicks under software rendering and reads the world clock from `/api/world` (a snapshot's tick stands still at night). Screenshots wait for interiors by frames, restart a crashed browser and skip the first-visit hint. +- **`/` is the town.** Overview, Live World, Traces, Experiments, Integrations and Settings stay; the mode rail replaces the run sidebar except on `/world` and `/traces`, and `/world?run=` shows "Agent District › {building} › Session #n" with **Back to town**. Old `/?run=` links redirect to `/world?run=`. +- The light-theme world decision is now numbered DL-013 (it was DL-009), so the plan of record's DL-009–012 stay valid. +- The tracer's `TraceSessionManager` takes an `onSessionSeen` hook and can close a session Runloop stopped (`closeStopped`, inferred); the recorder has `subscribeAll`. - Write guards (JSON content type, allowed Origin) apply to `DELETE` as well as `POST`. - Experiments is in the sidebar; the design system lists it as shipped. - The Metrics panel shows an executed run's tool calls by category, from Runloop's own tools (the mapping compare uses), instead of "not categorised". @@ -57,6 +82,8 @@ All notable changes to Runloop. The format follows [Keep a Changelog](https://ke - Studio URLs end in a slash (`/overview/`), so the static export works on plain static servers. ### Fixed +- **Immersive town review (2026-10-18), game feel and truth:** Add to World's last step follows the live agent (it hands over to the agent's own card on the first real event and never says "waiting for its first event" after one), and a link to a lot that does not exist or is taken stays in lot picking with the reason; a double click enters a building even when the second press lands on the card the first one opened (the town times double clicks itself); Explore, Agents, Intelligence and Lab, the World Log and compare leave the building the viewer is in instead of moving the camera away from a still-open interior; cards stand beside the screen box of what they describe (a building, an interior, a lot), stay above the minimap, and dock when nothing fits; the session strip and the guide banner centre in the area an open sheet leaves free (the strip rises above the minimap when it would be too narrow), and notifications start below the banner; on phones the card stands above the session strip; the strip says "Open · idle 12m" for a quiet session ("Live" only when the agent is live) and its picker says Open, not Live; a compact agent label keeps "Incident" or "Needs approval" in words; an agent going offline after its session closed is in the bell's list only (no "need you", no popup); a real agent's cue and its notification show together, and a stale "is live" or "started a session" never pops after newer facts; the HQ diorama is in fixed town materials (no longer a black slab in daylight or a white model in the light theme); night stars come out just above the haze line, so the lowest tilt shows them; a loading veil (the town's name and a progress line) covers the stage until the town is drawn, and the first-visit hint starts only then; a pinch that starts on the HUD zooms the town, never the web page; closing a card, the directory or the session strip gives focus back to what opened it. +- **World integration (2026-10-18):** the runtime now wires the agent layer and dispatch (they existed but were not mounted, so `/api/agents` answered 404); a cancelled or timed-out dispatch closes its traced run as interrupted (inferred) instead of reading live for 5 minutes; a dispatched session in a workspace that was traced with `--plugin-dir` is traced again (the plugin is passed unless `claude plugin list` reports it installed); keys typed right after ⌘K reach the palette instead of page shortcuts (focus no longer waits for an animation frame); a deep-linked `?focus=resident:` frames the resident once its position exists; at phone width the compact "Add" button is named "Add to World" for assistive technology; inside an agent's building the interior follows a session that starts while you watch; a grouped dispatch item in the feed shows the provenance of its latest event (the system finishing it, not you approving it). - `--doctor` now finds the plugin data dirs Claude Code actually uses (`runloop-tracer-inline` for `--plugin-dir`, and any other `runloop-tracer*` directory) when run from a shell. - The Studio no longer shows "Runtime offline" after a single slow health check. It needs two consecutive failures once it has been online. - In the 3D world, a selected incident label is no longer covered by the selected agent's label. A second required label shrinks before it may overlap. diff --git a/README.md b/README.md index c5eacc8..84ed1f1 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,17 @@ # Runloop -A local flight recorder for coding agents. Runloop records every event of an agent run in an append-only log on your machine, shows the run live as a small 3D world with a timeline and an inspector, replays any moment without executing anything again, and compares two runs to show what a change to your setup did. +**A world, not a dashboard.** Runloop is a small town that lives on your machine, and you step into it like a game. Simulated residents keep its streets busy; your real AI agents move in as buildings in the Agent District. When Claude Code works in a project you added, its building lights up, its robots go to work by kind of tool, a failing test shows as an incident, and every session can be replayed, traced and compared without leaving the town. -![A real Claude Code session in Runloop Studio: the main agent and two Explore subagents as robots at work, the timeline and the run's provenance](docs/assets/claude-code-session.png) +![Runloop Commons edge to edge: the town and the countryside around it, the clock and speed top-right, the minimap and the hotbar](docs/assets/town.png) -**Status: prototype.** It records two sources today: -- **Claude Code sessions**, traced through a plugin. Runloop records what Claude Code reports and never runs or approves anything itself. -- **A scripted demo mission**, executed by Runloop's own runtime with real tools. +**Status: prototype.** What it does today: +- **The town** (Runloop Commons): 80 simulated residents with needs, schedules and friendships, 28 buildings, a day and night that follow the simulated clock. It is deterministic and persistent: it pauses while Runloop is off and says so when it resumes. +- **A game you stay in:** the town fills the screen and the land runs on to the horizon; a free camera (pan, zoom, turn, tilt, follow, fly into buildings); a HUD with the clock, a minimap, a hotbar, cards next to what they describe and an Esc menu. Traces, compare, sessions, settings and Add to World all open inside the town. A calm, synthesized soundscape follows the camera; event sounds play only for real agent events. +- **Real agents:** Claude Code sessions, traced through a plugin. Runloop records what Claude Code reports and never runs or approves anything by itself. +- **Assign a task** to an agent (optional): Runloop shows the exact `claude -p` command, and runs it only after you approve it. +- **Runs, replay and compare:** every session's events, a run diorama, Traces, and Experiments to compare two runs. A scripted demo mission shows them without Claude Code. -Any two runs can be compared: pin one as a baseline, run your agent again, and **Experiments** shows what differed, with links to the events behind it. See [TASKS.md](TASKS.md) for what is next. +See [TASKS.md](TASKS.md) for what is next. ## Quickstart @@ -19,11 +22,71 @@ pnpm install pnpm dev # runtime API on http://127.0.0.1:4380, Studio on http://localhost:3000 ``` -Open http://localhost:3000 and click **Run demo mission**. Three scripted agents survey a small repo, implement a task store and run its test suite. The first test run really fails (10 passed, 2 failed). The Builder then patches the bug, and the rerun passes 12/12. Click a timeline row or a robot to inspect its events. Drag the replay bar to see any earlier moment. +Open http://localhost:3000: that is the town, edge to edge. Residents walk to work, eat, meet on benches and go home; the clock reads "Day 1 · 08:00" on a new world. Pause it, or run it at 1×, 4× or 16× (real agents always run at real speed). Click a resident to follow them and see what they are doing and **Why**: every reason links to the recorded decision event it came from. Double-click a building to fly inside; ⌘K / Ctrl-K finds anyone. -Then try compare: click **Pin as baseline** in the run header, run the demo mission again, and open **Experiments**. The demo is deterministic, so two demo runs make the same calls with the same results and differ only in timing; compare is meant for your own agent's runs (next section). +### Controls -Runs are stored in `.runloop/` at the repo root (set `RUNLOOP_DATA_DIR` to use another directory). They survive runtime restarts and replay from the log. +| | Mouse and touch | Keys | +|---|---|---| +| Move | drag · one finger | W A S D, arrows | +| Zoom (toward the pointer) | wheel · pinch | + − · 0 whole town | +| Turn · tilt | right-drag · two-finger twist; two fingers up or down tilt | Q E · Z X · R reset | +| Select · follow | click a resident or robot (it is followed) | F follow, Esc stop | +| Enter a building | double-click · double-tap | Enter | +| Modes | the hotbar | 1 Explore · 2 Agents · 3 Intelligence · 4 Lab · N Add to World | +| Views | the minimap's Map and List | M 2D map · L directory of everyone and everything · H hide the HUD | +| Menu | ☰ on the hotbar | Esc (backs out one step at a time, then opens the menu) · ? keyboard map | + +The Esc menu holds Display (theme, quality, motion, the 2D map instead of 3D), Sound, Controls, Runtime, Connections and links to every full page. With reduced motion the camera cuts instead of flying and the decorative life stands still; without WebGL the town is drawn as a 2D map with the same HUD. On a phone the town is a lighter 3D scene with touch gestures and cards as bottom sheets. + +**Sound** is synthesized in the browser (no audio files): birds and breeze by day, crickets at night, a murmur near the café and the fountain, distant traffic, and soft interface sounds. Four cues play only for real agent events (an agent goes live, needs approval, hits an incident, finishes a task), never for simulated activity, and each is also a notification. Nothing plays before your first click or key press; a hidden tab is silent. The speaker button (top-right) mutes it; Esc › Sound has the volumes and "reduce sound". + +The town and its log are stored in `.runloop/` at the repo root (set `RUNLOOP_DATA_DIR` to use another directory) and survive restarts. `RUNLOOP_WORLD=off` runs Runloop without the town. + +## Add Claude Code to the world + +1. Press **N** (or **Add to World** on the hotbar). The camera flies to the Agent District and the free lots light up with an outline and a label; click one (or its label button). A card on the lot asks for a name (for example "Engineering House") and the absolute path of a project folder. The building appears as an unpainted model: *waiting for its first event*. +2. Start Claude Code in that folder with the Runloop plugin (with the runtime running): + ```sh + claude --plugin-dir /path/to/runloop/integrations/claude-code-plugin + ``` + or install the plugin once from the marketplace (see [Trace Claude Code](#trace-claude-code)). +3. On the first prompt the building goes **live** (in about a second) and the Add to World card hands over to the agent's own card, which says its health as it is (live, idle, an incident, offline). The card shows what the session is doing by kind of tool, its subagents, and a failing command as an incident. **Enter building** shows the session's robots at their stations and a session strip: pick a session, scrub it (the diorama shows the recorded state at the playhead), open its **Trace** or **Compare** it, all inside the town. +4. When the session ends the agent reads *offline*; the town keeps living. Sessions from other folders wait under **Agents › Arrivals**, where you can claim them for an agent. + +| A live agent and its card | Inside the agent's building | +|---|---| +| ![The Engineering House agent live in the Agent District, its robot at the door and its card: Real · Claude Code · observed, Live, Enter building, Assign task, Trace](docs/assets/agent-live.png) | ![Inside the agent's building while a session runs: Claude and two Explore subagents at their stations, and the session strip with the replay bar at Live](docs/assets/agent-interior.png) | +| **Add to World: pick a lot in the town** | **A resident's Why, in a card next to them** | +| ![Add to World: the Agent District's free lots outlined, each with a label button, and the lot list in the banner](docs/assets/add-to-world.png) | ![A resident followed through the town, their card showing what they do and why, each reason linked to its event](docs/assets/resident-why.png) | +| **Assign task: a real dispatched run, done** | **Night in the Commons** | +| ![The Assign task dialog after a real read-only run: Claude Code's reported result, cost and output](docs/assets/assign-task.png) | ![The town at night: lit windows where residents are home, street lamps, the countryside in the dark](docs/assets/town-night.png) | + +| The horizon, tilted low | Intelligence: a trace beside the live town | +|---|---| +| ![The camera tilted down to the horizon: the town, farms, the river and forest fading into haze and mountains](docs/assets/town-horizon.png) | ![The trace sheet open over the right side of the town: a Claude Code session's timeline and a failed Read in the inspector](docs/assets/intelligence.png) | +| **Lab: compare two runs at the Lattice Lab** | **Inside the Corner Café** | +| ![The compare sheet beside the town: baseline and candidate sessions, outcome, tests, failures, duration, tool calls and tokens, with what was not reported](docs/assets/lab.png) | ![The café with its roof lifted: tables, the counter and residents inside, and its card with who is there](docs/assets/cafe-interior.png) | + +### What is real and what is simulated + +- **Residents are simulated.** They carry a ◇ *Simulated* label. Their decisions come from a rule-based engine (needs, schedules, traits), and their **Why** shows only the factors recorded in the decision event, each linked to it. No language model invents a reason. +- **Agents are real.** An agent building shows only what Claude Code's hooks reported (*Real · Claude Code · observed*) or what Runloop inferred from it, labelled *inferred*. An agent that has sent no event is never shown as live; unknown values read *not reported*. +- Every event in the town's log has a provenance: `simulation`, `user` (you), `external-agent`, `system` (Runloop) or `inferred`. The World Log (**Intelligence**) filters by it. +- The project folder's absolute path stays on your machine: the world, its events and the API only ever show the folder's name and a short hint. + +### Assign a task (dispatch) + +An agent's card has **Assign task**. Write the task, pick a preset and a budget, and **Preview command**: Runloop shows the exact `claude -p` command, model, tools, budget, timeout and the names of the environment variables it passes. **Approve in Runloop…** opens the runtime's own approval page in a window; it shows the same command, and nothing runs until you approve it there. The Studio follows the outcome and then shows the task's output. + +- **Presets:** *Read-only* (`Read Grep Glob`) or *Edit + test* (adds `Edit`, `Write` and `Bash(pnpm test:*)`), with `--permission-mode dontAsk`: any other tool is refused, never asked. +- **Limits:** haiku by default; the budget (default $0.25, at most $1) is enforced by Claude Code with `--max-budget-usd`; the timeout (default 10 min) by Runloop; at most one running task per agent and two in total. **Cancel** stops the whole process group. +- **Billed to your Anthropic account. Edits land in your working tree. "Done" is what Claude Code reports**; Runloop does not check the work. +- Approval happens only on the runtime's own page (`/approve/`: no CORS, unframeable, a short minimum review), with a single-use nonce bound to the exact command. No web page, the Studio included, can approve a task by itself. Local processes are trusted, as for the rest of the API (see Security notes). Every step is written to an audit log. `RUNLOOP_DISPATCH=off` turns dispatch off. + +### The demo mission + +Without Claude Code, open **Sessions** (Esc › Pages, ⌘K "Sessions", or `/overview`) and click **Run demo mission**; it opens in the town, in the trace sheet's World tab. Three scripted agents survey a small repo, implement a task store and run its test suite. The first test run really fails (10 passed, 2 failed); the Builder patches the bug and the rerun passes 12/12. Drag the replay bar to see any earlier moment. ## Trace Claude Code @@ -54,17 +117,13 @@ To check the setup, run `node integrations/claude-code-plugin/scripts/runloop-ho 1. Run a task with Claude Code and click **Pin as baseline** on that run (the run header, or the run history on Overview). 2. Change one thing (CLAUDE.md, a skill, a subagent, a hook, the model) and run the same task again. -3. Open **Experiments** (or **Compare** in the new run's header). The baseline defaults to your newest pin and the candidate to the newest other run of the same project. +3. Press **4** for the Lab (or **Compare** in the new run's header, or `/experiments`). The baseline defaults to your newest pin and the candidate to the newest other run of the same project. Experiments shows the outcome, the latest test results (fixed, newly failing, still failing), the files each run read or edited, tool calls by category, tokens and models per agent as reported, permission requests, turns and subagents, and the **first divergence**: the first tool call where the two runs went different ways. The test runs, failures, files, categories, usage reports and the divergence link to their events in Traces. A value one run did not report shows as *not reported* and is never compared; so does a token total that covers different agents in each run. -| Claude Code session | Failing test as an incident | -|---|---| -| ![A recorded session: Claude and two Explore subagents at work](docs/assets/claude-code-session.png) | ![The failing test run as an incident, with the inspector](docs/assets/claude-code-failure.png) | -| **Network view (2D)** | **Experiments: two real sessions compared** | -| ![Agent graph with delegation edges](docs/assets/claude-code-network.png) | ![Experiments comparing two real Claude Code sessions](docs/assets/experiments-compare.png) | +![A recorded Claude Code session in Live World: Claude and two Explore subagents at their stations, the timeline and the run's provenance](docs/assets/claude-code-session.png) -These screenshots show real `claude -p` sessions (Haiku, 2026-10-17) traced end to end and opened in the current Studio; see the [verification report](docs/status/verification-report.md). +This screenshot shows a real `claude -p` session (Haiku, 2026-10-17) traced end to end; the town images above come from real sessions on 2026-10-18. See the [verification report](docs/status/verification-report.md). **What is collected.** The plugin sends these to the local runtime only: - the session id; @@ -81,50 +140,129 @@ File contents, diffs, full command output and transcripts are never collected. W ## Commands ```sh -pnpm dev # runtime + Studio +pnpm dev # runtime + Studio (the town at http://localhost:3000) pnpm serve # runtime only (port 4380; --port to change it) pnpm demo # run the demo mission headless and print one trace line per event pnpm test # unit, integration and end-to-end tests (vitest) pnpm typecheck pnpm build # static export of the Studio (apps/studio/out) -pnpm screenshots # visual regression screenshots of every page (needs a runtime and Studio running) +pnpm screenshots # visual regression screenshots: the town (horizon, street, cards, the café, Add to World, the agent and its building, sheets, the Esc menu) and every page (needs a runtime and Studio running) +node scripts/e2e-world.mjs # the town lives: rendering, residents moving, ⌘K → Why, Follow, the café interior +node apps/studio/src/audio/check/render-check.mjs # the soundscape in headless Chromium (autoplay, levels, settings) +node scripts/perf-town.mjs # town render budgets (draw calls, triangles, JS per frame) under SwiftShader +pnpm --filter @runloop-world/sim bench # simulation tick cost and events per sim day ``` +Environment: `RUNLOOP_DATA_DIR`, `RUNLOOP_WORLD=off`, `RUNLOOP_DISPATCH=off`, `RUNLOOP_DISPATCH_ENV` (extra variable names passed to dispatched `claude`, e.g. `HTTPS_PROXY,NODE_EXTRA_CA_CERTS`), `RUNLOOP_TRACE=off`, `RUNLOOP_INGEST_TOKEN_FILE`, `RUNLOOP_ALLOWED_ORIGINS`; the runtime's usage text (`packages/runtime/src/cli.ts`) describes each. + ## How it works Three layers stay strictly apart: -1. **Execution truth.** The append-only event log. Tool results come only from tools that really ran (executed runs) or from what the source reported (traced runs). -2. **World simulation.** A pure function of the log and a world definition: where each robot is and what it is doing. -3. **Presentation.** The Studio draws layer 2 and never writes back to it. +1. **Execution truth.** The append-only event log of each run. Tool results come only from tools that really ran (executed runs) or from what the source reported (traced runs). +2. **World simulation.** Two pure functions: a run's diorama (where each robot is), and the town (residents, buildings, the clock), which has its own append-only log with a provenance on every event. +3. **Presentation.** The Studio draws both and never writes back to them; your commands (pause, open a building, Add to World) are recorded as `user` events. -Live view and replay fold the same events through the same reducers, so they always agree. More detail: -- [Architecture overview](docs/architecture/overview.md) +Live view and replay fold the same events through the same reducers, so they always agree. The town is deterministic: the same seed and commands give the same log, in any process. More detail: +- [Architecture overview](docs/architecture/overview.md) and [world simulation](docs/architecture/world-simulation.md) - [Event model](docs/architecture/event-model.md) - [Claude Code tracer design](docs/plans/claude-code-tracer.md) - [Design system](docs/design/design-system.md) - [Current state](docs/status/current-state.md) +## Where Runloop Is Going + +**Runloop is a living world, not just a window into agent execution.** + +The long-term goal is to build a persistent, autonomous society where simulated residents live and work, buildings serve real purposes, organizations pursue goals, and meaningful events emerge from interactions across the world. Real AI agents can enter this society as connected participants, contribute to tasks in their actual environments, and bring their outcomes back into the world. + +The world should remain meaningful even when no external agents are connected. Traces, replay, and observability are important capabilities, but they are not the ultimate product. They are tools for understanding and improving what happens inside a larger, evolving system. + +### The World We Want to Build + +- **A living society:** Residents have identities, needs, routines, goals, memories, and relationships that influence their decisions and the world around them. +- **Functional places:** Buildings are more than visual structures. Workplaces, laboratories, offices, and shared spaces have capabilities, resources, occupancy, and activities that shape what can happen there. +- **Organizations with purpose:** Teams and organizations work toward objectives, manage dependencies, encounter constraints, and respond to changing circumstances. +- **Real agents as participants:** External AI agents connect through explicit integrations, occupy workspaces in the world, perform authorized tasks in their real environments, and contribute observable outcomes. +- **A world that evolves:** The simulation maintains its own history and continues independently of external agent sessions, creating situations that users can explore, understand, and influence. +- **Experimentation through alternate worlds:** Users can checkpoint and fork a world, change conditions or rules, and compare outcomes to understand how different decisions affect the system. + +### From Agent Monitoring to a Living System + +The current prototype establishes the foundations: a persistent simulated town, resident routines, a Claude Code integration, provenance-aware event history, controlled task dispatch, and execution replay and comparison. + +The next challenge is to connect these capabilities into one coherent system. + +| Area | Current foundation | Direction | +|---|---|---| +| Society | Residents with needs, schedules, traits, and friendships | Meaningful goals, memory, consequential relationships, and social dynamics | +| Buildings | Spatial structures and some interiors | Functional rooms, capabilities, organizations, and shared resources | +| Agent ecosystem | Claude Code integration | A stable adapter contract and support for additional agent runtimes | +| Collaboration | Sessions and controlled task dispatch | Delegation, dependencies, handoffs, and coordination policies | +| World autonomy | Deterministic resident simulation | Explainable society-level goals and evolving situations | +| Experimentation | Replay and run comparison | World checkpoints, forks, altered conditions, and outcome comparison | +| Scale and reliability | Initial town and population benchmarks | Larger-world validation, performance budgets, and robust recovery | +| Product experience | Immersive local prototype: a full-screen 3D town with a game HUD, where every action opens inside the world | Better onboarding, accessibility, and sustained user validation | + +These are product directions, not claims that every capability is already implemented. See [TASKS.md](TASKS.md) and the [current-state report](docs/status/current-state.md) for the implementation status of individual milestones. + +### The Core Experience + +The experience we are working toward is a world that users can: + +1. **Explore** — enter a populated society and discover what residents, organizations, and connected agents are doing. +2. **Understand** — investigate events, decisions, dependencies, and the evidence behind what happened. +3. **Influence** — change conditions, create functional places, establish objectives, and shape how the world develops. +4. **Connect** — invite a real agent to perform an explicitly authorized task in its actual environment. +5. **Experiment** — fork a world, test alternative conditions, and compare the resulting behavior. +6. **Return to** — revisit a persistent world with meaningful history, new developments, unresolved problems, and opportunities for action. + +### Principles That Guide Development + +- **World first, traces second.** The society must have purpose and activity independent of agent-monitoring sessions. +- **Meaningful behavior over visual activity.** Residents should act for understandable reasons, not simply move to make the world appear alive. +- **Simulation and reality stay distinct.** Simulated events, externally reported activity, executed actions, and inferred information must have clear provenance. +- **Real actions require explicit boundaries.** External execution must respect permissions, approval policies, resource limits, and auditable outcomes. +- **Explainability and reproducibility matter.** Users should be able to understand why a simulated event occurred and reproduce or compare outcomes where appropriate. +- **Function before spectacle.** Visual design and motion should reveal activity, relationships, and cause and effect—not substitute for functional depth. +- **The world belongs to the user.** Persistence, recovery, data boundaries, and accessible ways to navigate and inspect the world are core product concerns. + +### What Comes Next + +The near-term priority is to make the world functionally meaningful before expanding its visual scope or integration count: + +1. Complete functional buildings, organizations, and shared capabilities. +2. Introduce explainable project goals, dependencies, and society-level activities. +3. Demonstrate an end-to-end workflow in which an authorized real agent performs a task and its reported outcome is reconciled with the world. +4. Implement world checkpointing and experimentation without replaying real-world actions. +5. Validate that users can understand the world, accomplish useful work, and find reasons to return. + +**The measure of progress is not how many residents populate the town or how many agents Runloop can trace. It is how much meaningful activity the world supports, how well users can understand and influence it, and how coherently simulated society and real AI systems work together.** + ## Security notes - The runtime listens on loopback only. Ingest from the plugin needs a per-user token (`~/.config/runloop/ingest-token`, created by the runtime) and rejects browser requests. - **Read endpoints are not authenticated.** Any local process of yours can read recorded runs, including prompts and commands, just as it can read the database file. +- **Dispatch** starts `claude -p` only after you approve the exact command on the runtime's own approval page (see [Assign a task](#assign-a-task-dispatch)); its edits are real and its cost is billed to your account. - In the demo, `run_command` executes the workspace's tests as your user, without OS-level isolation. That is safe for the fixed demo code. Read the sandbox notes in [packages/runtime/README.md](packages/runtime/README.md) before connecting an adapter that writes its own code. ## Repository layout ``` -apps/studio Next.js Studio: Overview, Live World (3D + Network), Traces, Experiments, Integrations, Settings -packages/events event schema, parsing, redaction, SequenceBuffer, HTTP/SSE and ingest contract +apps/studio Next.js Studio: the town (/) with its HUD, sheets and soundscape; the full pages Overview, Live World (3D + Network), Traces, Experiments, Integrations, Settings +packages/events event schema, parsing, redaction, SequenceBuffer, HTTP/SSE and ingest contract; world contracts (world.ts) +packages/town the town file format: schema, compile, raster, navigation fields (pure) +packages/sim the town simulation: residents, decisions, applyEvent, explain, replay (pure) packages/core layer 1: execution reducer, selectors, text formatting, run compare (summarizeRun, diffRuns) packages/world-engine layer 2: world definition, grid paths, motion, character intents packages/replay combined reducer and ReplayIndex (state at any sequence or time) -packages/runtime local runtime: SQLite recorder, tool sandbox, run manager, Claude Code ingest, HTTP + SSE API, CLI +packages/runtime local runtime: SQLite recorder, tool sandbox, run manager, Claude Code ingest, world host, agents, dispatch, HTTP + SSE API, CLI packages/testkit fixtures and helpers for tests integrations/claude-code-plugin the Claude Code plugin (observe-only hooks, spool, --doctor) -worlds/research-campus the default world (rooms, stations, presets) +worlds/runloop-commons the town: town.json and 15 interiors +worlds/research-campus the run diorama (rooms, stations, presets); also the agent headquarters' interior examples/coding-mission the demo mission and its example repo -scripts/screenshots.mjs visual regression screenshots (Playwright) +scripts/ screenshots.mjs, e2e-world.mjs (Playwright), perf-town.mjs (render budgets) docs/ strategy, architecture, plans, design system, status ``` diff --git a/TASKS.md b/TASKS.md index 8312707..bb4c416 100644 --- a/TASKS.md +++ b/TASKS.md @@ -4,6 +4,40 @@ A task is **Verified** only when its acceptance criteria were exercised and the evidence is recorded in `docs/status/current-state.md` or `docs/status/verification-report.md`. +## Milestone I: The immersive town (current, DL-014) + +Founder direction (2026-10-18): "it should look like an immersive game … the user should be able to do everything while being inside the simulation." Experience contract written first, then built by four owners (WORLD, HUD, OVERLAYS, SOUND) and integrated on 2026-10-18. Evidence: `docs/status/verification-report.md` §9. + +| ID | Task | Status | Acceptance criteria and evidence | +|---|---|---|---| +| I1 | **One place:** `/` is the world edge to edge (no sidebar, no top bar); land to the horizon (outskirts, fields, river, lake, forest, hills, mountains), haze and a sky from sim time | Verified | The town canvas equals the viewport at 1920×1080, 1440×900 and 1280×800; at tilt 10° the horizon, mountains and haze show (screenshots). Land adds 13–15 draw calls (budget 16) and 0.2 ms JS; every perf bookmark within budget. | +| I2 | **Free game camera:** perspective with a narrow FOV, pan with inertia, zoom to the cursor, rotate, tilt 10°–62°, WASD/QE/ZX/R/0, soft bounds, follow, fly-in with the roof lifting, `cam=` links with az and tilt | Verified, with gaps | `cameraMath`/`cameraController` unit tests; in the browser: drag pans, wheel zooms, E turns 45°, right-drag rotates, tilt clamps at 10° and 62°, W/Z held and R work, 3- and 5-part `cam=` links, ⌘K follows a resident, double-click/Enter flies into the café. On phones, entering a building frames its diorama (it zoomed out before this pass's fix). **Gap:** measured under SwiftShader at 1–2 fps, so held keys and flights run at that frame rate; no real-GPU timing. | +| I3 | **Game HUD:** world chip, Find, bell, clock and speed with sound, notifications, minimap, hotbar, anchored cards, guide banner, Esc menu; fades while dragging, H hides | Verified | Journey: the HUD fades while dragging (not under reduced motion), the Esc cascade ends in the menu, its Sound tab has four sliders, Connections shows the Integrations content; 138 screenshots at 1440 and 390 in both themes, day and night, looked at; the canvas equals the viewport at 1920, 1440 and 1280. | +| I4 | **Everything in the world:** Add to World on a lot in 3D, Assign task from the agent card, Intelligence (World Log + trace sheet), Lab (compare sheet), Sessions, the building's session strip, ⌘K; legacy routes as full pages | Verified | Journey on a real runtime: N → lot picking → card on the lot → the HQ "waiting for its first event"; a sanitized Claude Code fixture through the real plugin hook made the agent live; inside its building the strip scrubbed to "Replay · recorded state" and Go live returned; 3 → World Log, a run link opened the trace sheet on `/` while the clock kept running; 4 → compare sheet; the demo started from the Sessions sheet in the world (10/2 at #36, 12/12 at #55); `/traces`, `/experiments`, `/world`, `/overview`, `/settings`, `/integrations` render as before; `/?run=` still redirects; `/?overlay=trace&run=&event=` opens the sheet. | +| I5 | **Ambient decorative life** (cars, birds, clouds, windmill): never pickable or listed, ≥ 12 cells from agent lots, still when paused, off under reduced motion | Verified | `geometry/surroundings.test.ts` (distances, never pickable); WORLD's browser check (still when paused and under reduced motion). | +| I6 | **Sound:** Web Audio bed following the camera, UI sounds, cues only for live real agent events, autoplay and hidden-tab handling, settings | Verified by measurement; not listened to | Unit tests (cue table over every event type, limiter, mix, settings, engine); `render-check.mjs` 51/51; in the town: no AudioContext before a gesture, the bed's layers follow the listener (inside the café: interior), a real agent's fixture session played `agentLive`, `needsApproval` and `incident` (repeats held by the cooldown), no cue from the simulated events around it, a hidden tab suspends; at street level by a busy café patio the murmur rose to 0.81 and the wind fell to 0. **Gap:** nobody has listened to it; Safari untested. | +| I7 | **Phones:** a light 3D town, touch gestures, bottom-sheet cards | Verified in emulation | 390×844 with touch: the 3D town renders, the hotbar fits, a one-finger drag pans, the agent card is a bottom sheet, entering a building frames its diorama above the session strip. No real phone or frame time. | +| I8 | **2D routes and access:** directory (L), WebGL-off map with the same HUD and Add to World as a dialog, reduced motion (cuts, no fades, no whoosh), keyboard | Verified, with gaps | Routes run: L opens the directory tree; WebGL off draws the map with the hotbar and the Add to World dialog with its lot list; reduced motion: no HUD fade, no whoosh, the turn is a cut (no value in between in 26 samples; without reduced motion 14 of 17 samples were in between). **Gap:** no axe or screen-reader pass. | +| I9 | **Truth fixes carried from the review** (approval on the runtime's page, Real chip only for registered agents, "Task running · no events yet", clock re-base and `wallMs`, world-scoped resume, compare caveats, cost) | Verified | Unit tests (clock, store reset, selectors, compare wording, approval outcomes); the approval flow up to Decline on the runtime's own page in the browser: nothing ran. | +| I10 | **Review fixes (game feel, truth, access):** the 17 findings of the immersive review (Add to World's last step, double click, modes leaving a building, HUD layers and card placement, phone pinch and strip, honest "Open · idle" strip and compact alert labels, offline is not "need you", cue and notification together, the diorama's materials, night stars, loading veil, focus return, the notes vs the banner, lot links) | Verified | 23 new unit tests (`world/reviewFixes.test.ts`, `hud/reviewFixes.test.ts`, `overlays/stripMode.test.ts`, `town/input.test.ts`, `town/hqPalette.test.ts`, `status.test.ts`); browser checks on a runtime with fixture sessions through the real hook (verification report §10). **Gap:** frame-rate-bound timings (SwiftShader). | + +**Open:** frame time on a real GPU and on a real phone; listening to the sound (and Safari); glass contrast in the light theme over a bright sky (not measured); two WebGL canvases (the trace sheet's World tab) on real hardware. + +## Milestone W: A world, not a dashboard + +Vision: `docs/product/world-vision.md` (DL-009). Plan of record: `docs/architecture/world-simulation.md` (DL-010–012). Slice S0–S4 built on 2026-10-18; evidence in `docs/status/verification-report.md` §7. + +| ID | Task | Status | Acceptance criteria and evidence | +|---|---|---|---| +| W1 | **A world that exists** (S0 contracts, S1): the Commons town file, compile and navigation, `WorldHost` (clock, persistence, checkpoints, resume), `/api/world` and SSE, the Studio's town, clock, map and directory | Verified | Close and reopen: the same world, buildings and identities return. **Evidence:** automated S1 tests (compile, door reachability and BFS paths, 40 ticks in 10 s at 4×, pause and resume, restart and SIGKILL resume, SSE fold equals the server hash, legacy links). Real runtime: after a SIGTERM and restart the world id, content hash, seed, creation time and roster (sha `2653ce58edfeca18`, 80 residents) were identical and `world.resumed_after_offline` was appended. Screenshots: town, district and street, both themes, four widths. Not measured: frame rate on a real GPU. | +| W2 | **A population that lives** (S2): needs, schedules, decisions, interactions, closing hours, Why, World Log, ⌘K | Verified, with gaps | Residents act without clicks, visit valid destinations, react to change, and every transition can be explained from recorded state. **Evidence:** a sim day at 80 and 120 residents meets the action, sleep and eat targets, nobody awake goes 6 h without deciding, the liveliness floor holds, closing the café re-routes walkers in the same tick, replayed `explain` equals live. `scripts/e2e-world.mjs --fresh`: 62 of 80 residents walking within 3 s (needed ≥ 48), the clock advancing, ⌘K → Why with every factor citing a recorded `resident.decided`, Follow, the café interior (11/11). Three residents were followed through a recorded lunch: every Why named what they then did. **Gaps:** the lunch was followed in the log, not watched at 4× in the browser; event volume is over the plan's budget (~4.8k events, 1.6 MB per sim day at 120 residents vs ≤ 6k events and ≤ 1 MB at 300); the Commons houses 136, so the 300-resident budgets were measured on a bench-only 288-resident variant. | +| W3 | **Functional buildings and organisations** (teams, tasks, rooms with capabilities) | Not started | Cut from this slice (plan §11). | +| W4 | **A real agent joins** (S3 observe, S4 dispatch): Add to World, binding, health, capabilities, the agent building and interior, Arrivals and claim, Assign task with approval | Verified, with gaps | A real Claude Code agent's confirmed activity shows in its building; simulated activity cannot be mistaken for real; the world keeps living while the agent is offline. **Evidence:** a real `claude -p` session (haiku, $0.016) in a folder added through the UI went live 1.1 s after its first hook, showed tool counts by category and a subagent, read offline after SessionEnd while the town kept its clock; two real dispatches through the Assign task dialog ($0.0023 and $0.0015) and two more watched from inside the building ($0.0036 and $0.0033) completed, the interior showing Claude reading at the Observatory; after a restart the agent's building survived, read offline, and its sessions replayed. Automated: S3 and S4 acceptance tests (P4, P5, P7, P8, P11, nonce, Origin, cancel, audit) and a wiring test through `createRuntime`. **Gaps:** the real session piped `node --test` through `tail`, so no incident appeared live (the incident path is covered by ingesting a recorded real session over HTTP); subagent helpers inside the building were seen only in replay (the read-only dispatches use no subagent); a real cancel at the first tool call was not run (fake `claude` only); no interactive (non `-p`) session. | +| W5 | **The world becomes a lab** (forks, history scrubber, re-run) | Not started | The world log and `WorldReplay` exist; the UI is plan §11 (W5). | +| W6 | **Emergent behaviour and scale** (LLM residents) | Not started | Plan §11 (costs money). | + +**Open follow-ups from the slice:** reduce `resident.decided` volume (or trips) to meet the event budget; a 300-resident town; measure frame time on a real GPU (plan §9); onboarding coachmarks, axe and keyboard-only passes; an agent offline after a normal session end counts as "needs you" in the ticker (per the UX draft; may be too loud). + ## Milestone A: Foundation | ID | Task | Status | Evidence / location | @@ -63,7 +97,7 @@ Spec: `docs/design/design-system.md` (founder brief). This milestone runs in par | D7 | Search and filter in the timeline and logs (agent, tool, category, status, text) | Verified | Filters compose; URL-addressable; tested | | D8 | Run metrics panel (duration, calls by category, failures, tokens and cost where reported) | Verified | Defined calculations; "not reported" never shown as zero | | D9 | Visual regression screenshots of every page and state | Verified | Script in the repo; images checked on each milestone | -| D10 | World richness pass: the world fills its panel (tighter framing, collapsible timeline), robots read clearly at default zoom, warmer key/rim lighting and materials, restrained zone accent light; keep the Signature restraint | Verified (except the collapsible timeline, moved to D11) | At 1440×900 a robot is ≥ 28 px tall at default zoom; the board fills ≥ 80% of the world panel width; founder-reference look (diorama, chibi robots) recognisable in screenshots of both themes. **Evidence:** measured with the framing code at the stage sizes of real windows: at 1440×900 the board spans 97.0% of the stage and a robot projects to 44 px; at 1280, 96%; at 1920, 81–82%. Tests assert ≥ 80% and ≥ 28 px. Screenshots in both themes (verification report §6); light-theme look recorded as DL-009. | +| D10 | World richness pass: the world fills its panel (tighter framing, collapsible timeline), robots read clearly at default zoom, warmer key/rim lighting and materials, restrained zone accent light; keep the Signature restraint | Verified (except the collapsible timeline, moved to D11) | At 1440×900 a robot is ≥ 28 px tall at default zoom; the board fills ≥ 80% of the world panel width; founder-reference look (diorama, chibi robots) recognisable in screenshots of both themes. **Evidence:** measured with the framing code at the stage sizes of real windows: at 1440×900 the board spans 97.0% of the stage and a robot projects to 44 px; at 1280, 96%; at 1920, 81–82%. Tests assert ≥ 80% and ≥ 28 px. Screenshots in both themes (verification report §6); light-theme look recorded as DL-013. | | D11 | Collapsible timeline on Live World, so the world can take the full height | Not started | The world panel grows when the timeline collapses; state survives a reload; keyboard reachable | ## Milestone E: Generalisation diff --git a/apps/studio/next.config.ts b/apps/studio/next.config.ts index b16d6b5..4fddf3f 100644 --- a/apps/studio/next.config.ts +++ b/apps/studio/next.config.ts @@ -14,6 +14,8 @@ const nextConfig: NextConfig = { '@runloop-world/core', '@runloop-world/events', '@runloop-world/replay', + '@runloop-world/sim', + '@runloop-world/town', '@runloop-world/world-engine', ], turbopack: { diff --git a/apps/studio/package.json b/apps/studio/package.json index 14dad15..43d4500 100644 --- a/apps/studio/package.json +++ b/apps/studio/package.json @@ -16,6 +16,8 @@ "@runloop-world/core": "workspace:*", "@runloop-world/events": "workspace:*", "@runloop-world/replay": "workspace:*", + "@runloop-world/sim": "workspace:*", + "@runloop-world/town": "workspace:*", "@runloop-world/world-engine": "workspace:*", "geist": "1.8.1", "next": "16.4.2", diff --git a/apps/studio/src/app/page.tsx b/apps/studio/src/app/page.tsx index 739788a..85a80fd 100644 --- a/apps/studio/src/app/page.tsx +++ b/apps/studio/src/app/page.tsx @@ -1,15 +1,9 @@ -'use client'; -import { useRouter } from 'next/navigation'; -import { useEffect } from 'react'; -import { parseDeepLink, routeForDeepLink } from '../state/deeplink'; +import type { Metadata } from 'next'; +import WorldHome from '../components/world/WorldHome'; -/** `/` redirects: deep links (?run=&view=…) to their page, everything else to the Overview. */ -export default function RootRedirect() { - const router = useRouter(); - useEffect(() => { - const params = new URLSearchParams(window.location.search); - const query = params.toString(); - router.replace(`${routeForDeepLink(parseDeepLink(params))}${query ? `?${query}` : ''}`); - }, [router]); - return null; +export const metadata: Metadata = { title: 'Town' }; + +/** `/` is the town (plan §5). Old run links to `/` are sent on to their page by WorldHome. */ +export default function Page() { + return ; } diff --git a/apps/studio/src/audio/README.md b/apps/studio/src/audio/README.md new file mode 100644 index 0000000..20de849 --- /dev/null +++ b/apps/studio/src/audio/README.md @@ -0,0 +1,140 @@ +# `audio/`: the town's soundscape + +Plan of record: the immersive experience contract §4b (2026-10-18). Everything is +synthesized with the Web Audio API. There are no audio files, no samples, no new dependencies, and no licences. +The sound is calm by default and follows the camera. Event sounds play **only for real agent events**, so sound +obeys the same truth rule as motion. + +## API (`index.ts`) + +```ts +import { attachWorldAudio, getAudio, SoundSettings, SoundToggle, useAudioSettings, classifyCue } from '../audio'; + +getAudio(): RunloopAudio // the page's engine; an SSR-safe no-op on the server / without Web Audio + .state // 'locked' (before the first gesture) | 'running' | 'suspended' | 'unsupported' + .unlock() // from a user gesture; idempotent (the page's first pointer/key gesture calls it) + .ui(sound) // 'hover' | 'select' | 'open' | 'close' | 'menu' | 'confirm' | 'error' | 'toggle' + .whoosh(durationMs, pan?) // a camera fly (not for cuts; never under reduced motion) + .cue(cue, { agentId?, pan? }) // 'agentLive' | 'incident' | 'taskDone' | 'needsApproval'; false = not played + .setListener(l) // the town's TownListener (~4 Hz); the bed follows it + .setReducedMotion(v) + .getSettings() / .update(p) / .subscribe(fn) +attachWorldAudio(store): () => void // world store → cues for live real agent events (call once on `/`) +classifyCue(event, prevAgentState) // pure: which event earns which cue +useAudioSettings(): [settings, update] + // the HUD speaker toggle (icon + aria-pressed + "Sound on/off") + // the Esc menu's Sound tab: on/off, master/ambient/UI/events, reduce sound +``` + +### Wiring on `/` (HUD owns these files) + +| Where | Call | +|---|---| +| `components/world/hud/sound.ts` | `export function uiSound(name: UiSound) { getAudio().ui(name); }` (and `export type { UiSound } from '../../../audio'`) | +| `WorldHome.tsx` | `useEffect(() => attachWorldAudio(store), [store]);` | +| `WorldHome.tsx` → `TownStage` | `onFly={(ms, pan) => getAudio().whoosh(ms, pan)}` and `onListener={(l) => getAudio().setListener(l)}` | +| `WorldHome.tsx` → `WorldHud` | `sound={}` (clock widget and phone bar) and `soundSettings={}` (Esc menu) | + +Nothing else is needed. The first pointer or key gesture on the page unlocks sound, the tab's visibility suspends it, +and `attachWorldAudio` also passes the store's reduced-motion flag and the agent buildings and lots, which are never a murmur +source. With WebGL off (WorldMap) there is no listener: the bed stays silent and the UI sounds and cues still play. + +## Truth rule (P10 for sound) + +- `classifyCue` returns a cue only for `agent.*` events. It never returns one for `prov: 'simulation'`, and never for + residents, interactions, buildings, the clock or commands. This is tested over every `WORLD_EVENT_TYPES` entry and a recorded engine day. +- `attachWorldAudio` cues only **live** events. Nothing from the snapshot, the backfill after a reconnect, or the + 3 s after any resync or return to `live` makes a sound, and a disabled world never cues. It keeps its own agent state + (health, attention, open incidents), seeded from the snapshot, so a cue means "this just changed". +- Every cue repeats a notification with icon and text. A sound never stands alone. +- The ambient murmur follows **simulated** occupancy (café, fountain, diner, tavern, bakery, squares). It is + decorative ambience and never comes from an agent building. Settings and the README say so. + +| Event (vs the agent's previous state) | Cue | Sound | +|---|---|---| +| `agent.connection_changed`: health becomes `live` | agentLive | sine C5 + G5, 900 ms release, reverb | +| attention becomes `permission_requested` / `dispatch_awaiting_approval` | needsApproval | two FM bell knocks, 220 ms apart | +| attention becomes `incident`; `agent.activity_observed` open incidents rise | incident | low thump, then soft G4 → E4 | +| `agent.dispatch_changed` finished `completed` / failed, timed out, over budget, lost | taskDone / incident | C5 E5 G5 C6 arpeggio | +| `agent.session_changed` ended `succeeded` / `failed` | taskDone / incident | | + +Priority within one event: needsApproval > incident > taskDone > agentLive. The pacing is done by `CueLimiter`, which is pure +and has an injected clock: +- **Spacing and queue:** at least 1.2 s between cues; a queue of at most 2, strongest first; a queued cue expires after 4 s. +- **Rate:** at most 6 cues per minute. +- **Per agent and cue:** a cooldown of 60 s for agentLive, 30 s for needsApproval, 20 s for taskDone and 15 s for incident. +- **Silence:** nothing plays while the tab is hidden (and nothing is kept for later), nor for 3 s after a resync. + +## Graph and synthesis + +``` +ambient layers → outdoor (−9 dB inside) → outdoorLp (700 Hz inside) → ambient ┐ +birds → outdoor; birds → birdsSend (0.25) → reverb │ +UI voices → ui → uiSend (0.1) → reverb ├→ master (−1 dB headroom) → compressor (−18 dB, 3:1) → out +cue voices → events → eventsSend (0.2) → reverb (generated 1.6 s impulse) ─────┘ +``` + +- **Bed** (`ambient.ts`). Each layer is set by the mix rules in `mix.ts` from the listener (zoom01, night, paused, interior, sources): + - breeze: band-passed pink noise, swept slowly; + - high wind: gusts; + - birds by day: one-oscillator chirp calls, at a Poisson rate set by greenery and zoom; + - crickets by night: 18 ms pulse trains; + - murmur: two speech bands gated into syllables, panned to its source; + - water: grains and a rumble; + - distant traffic: brown noise under 140 Hz, plus an occasional pass-by by day; + - room tone inside a building. + + A paused world fades out its murmur, traffic, birds and crickets in about 1.5 s and keeps half the breeze. A 250 ms timer schedules 0.6 s + ahead on the audio clock. There are 40 persistent nodes in all, at most 12 one-shot voices (the oldest UI voice is stolen), and no + ScriptProcessor or AudioWorklet. +- **Lifecycle** (`engine.ts`): + - No AudioContext exists before the first gesture, and none is created while sound is off. The bed fades in over 2 s. + - Hidden tab: the master goes to 0 in 150 ms, the context suspends at 300 ms, and when the tab shows again it resumes and fades in over 600 ms. + - With no town on screen for 5 s, the context idles until the next sound. +- **Reduce sound** turns off the ambient bed and the hover ticks, lowers the UI by 6 dB, and keeps only the needsApproval, incident and taskDone cues. Reduced motion turns off the whoosh. +- **Settings** are stored in localStorage `runloop.audio.v1` (try/catch, clamped) and sync across tabs. Defaults: on, master 0.6, ambient 0.5, UI 0.4, events 0.7. + Sliders use a squared taper (half = −12 dB). + +Measured output at the default settings (offline render, headless Chromium; see below): + +| Sound | Level | +|---|---| +| Bed (RMS) | about −36 to −38 dBFS by day outdoors, −41 at night, −46 inside a building, −45 paused | +| UI clicks (peak) | −27 to −37 dBFS; the hover tick is −40 | +| Cues (peak) | −19 to −22 dBFS | +| Full volume | no single sound passes −5 dBFS; the busiest moment the pacing allows peaks at −2.2 dBFS | + +Nobody has listened to these sounds yet: every check is a measurement, and the levels are conservative. + +## Debug hook + +`window.__runloopAudio` is a read-only getter, available in dev and prod. It reports: +- `state`, `created`, `gestureSeen`, `hidden`, `idle`; +- `settings`; +- `listener`: freshness, level, zoom01, night, paused; +- `layers`: each layer's smoothed level, with `interior` and `passing`; +- `active`: the layers audible now; +- `buses`: the current gains; +- `voices`, `persistentNodes`, `cuesWaiting`; +- `lastCues`: each cue with its agent and its result (`play`, `queued`, `cooldown`, `rate`, `suppressed`, `hidden`, `off`, `reduced`); +- `lastUi`. + +## Verification + +``` +pnpm vitest run --project studio src/audio +node apps/studio/src/audio/check/render-check.mjs # headless Chromium; exit 1 on failure +node apps/studio/src/audio/check/render-check.mjs --serve # the sound board on :3074 (listen; ?ui=1 shows the toggle and settings) +``` + +- **Unit tests:** + - `cues.test.ts`: the S2 table over every event type and a recorded day, the four cues, and S3 limiter pacing, cooldowns, suppression and hidden; + - `mix.test.ts`: the mix rules; + - `settings.test.ts`: persistence and clamping; + - `engine.test.ts`: S1 lifecycle with a fake AudioContext, voices, hidden and mute, reduce, the bed and idling, the page hooks; + - `world.test.ts`: live-only cues, resyncs, reconnects and the real limiter. +- **Browser check** (`check/`): run live with the real autoplay policy, it checks for no context before a gesture, then a click + starts sound, the bed follows the listener, and a hidden tab suspends it. Rendered offline (OfflineAudioContext), it + checks that every sound is non-silent and peaks under 0.9 at default and full volume, that the bed RMS stays within bounds, + that the busiest moment does not clip (worst of 5 seeds), and that master, ambient, UI, events, mute and reduce each + change the output. `?layers=1` on the sound board renders each layer of each scene alone, for tuning the trims. diff --git a/apps/studio/src/audio/ambient.ts b/apps/studio/src/audio/ambient.ts new file mode 100644 index 0000000..6d57f9a --- /dev/null +++ b/apps/studio/src/audio/ambient.ts @@ -0,0 +1,338 @@ +/** + * The ambient town bed (contract §4b): breeze, high wind, birds (day), + * crickets (night), murmur near busy places, water, a distant traffic hum, + * and room tone inside buildings. It follows the camera through the mix + * (`mix.ts`); `update` is called about four times a second and schedules + * what happens in the next `LOOKAHEAD` seconds on the audio clock, so a late + * timer never leaves a gap. All synthesis: looping noise buffers, filters, + * a few oscillators. + * + * Persistent nodes: 28 here + 12 in the graph = 40 (the contract's cap). + */ +import type { AudioGraph } from './graph'; +import { OUTDOOR_INSIDE_HZ, OUTDOOR_OPEN_HZ } from './graph'; +import { INTERIOR_OUTDOOR_GAIN, smoothToward, type AmbientMix } from './mix'; +import { between, createPan, type Rng } from './synth'; + +/** How far ahead events are scheduled (s). */ +export const LOOKAHEAD = 0.6; +/** How fast levels follow the mix (s): a paused world fades out in ~1.5 s. */ +export const LEVEL_TAU = 0.5; + +/** + * Each layer's gain at level 1, calibrated with `render.ts` (`?layers=1` on + * the sound board prints each layer alone) so the default bed stays quiet + * (about −37 to −40 dBFS RMS by day) and no layer dominates. + */ +export const TRIM = { + breeze: 1.2, + wind: 0.96, + birds: 0.2, + crickets: 0.11, + murmur: 0.9, + water: 2.2, + /** The water's low rumble, relative to its grains. */ + rumble: 0.25, + traffic: 0.4, + /** The mix's room level (0.08), heard through the interior's −9 dB, kept under the voices. */ + room: 0.5 / INTERIOR_OUTDOOR_GAIN, +} as const; + +export const SOUNDSCAPE_NODES = 28; + +export interface BedLevels { + breeze: number; + wind: number; + /** Calls per second. */ + birds: number; + crickets: number; + murmur: number; + water: number; + traffic: number; + room: number; + interior: boolean; + /** A distant car is passing. */ + passing: boolean; +} + +type Levels = { breeze: number; wind: number; birdsRate: number; crickets: number; murmur: number; water: number; traffic: number; room: number; day: number }; +const LEVEL_KEYS = ['breeze', 'wind', 'birdsRate', 'crickets', 'murmur', 'water', 'traffic', 'room'] as const; + +export type SpawnBird = (t: number, species: number, pan: number, peak: number) => void; + +export class Soundscape { + private readonly sources: AudioScheduledSourceNode[]; + private readonly breezeGain: GainNode; + private readonly lfo: OscillatorNode; + private readonly windGain: GainNode; + private readonly cricketOsc: OscillatorNode; + private readonly cricketEnv: GainNode; + private readonly cricketPan: AudioParam | null; + private readonly murmurGain: GainNode; + private readonly murmurPan: AudioParam | null; + private readonly waterGain: GainNode; + private readonly rumble: GainNode; + private readonly waterPan: AudioParam | null; + private readonly trafficLp: BiquadFilterNode; + private readonly trafficGain: GainNode; + private readonly roomGain: GainNode; + + private L: Levels = { breeze: 0, wind: 0, birdsRate: 0, crickets: 0, murmur: 0, water: 0, traffic: 0, room: 0, day: 1 }; + private pans = { murmur: 0, water: 0 }; + private gust = 0.6; + private next = { gust: 0, syllable: 0, grain: 0, train: 0, call: 0, lfo: 0, pass: 0 }; + private pass: { start: number; dur: number } | null = null; + private cricketVoice = 0; + private murmurOn = false; + private waterOn = false; + private birdsOn = false; + /** Inside a building (the outdoor bed is muffled). */ + inside = false; + + constructor( + private readonly g: AudioGraph, + private readonly rng: Rng, + private readonly spawnBird: SpawnBird, + ) { + const { ctx, buffers } = g; + const loop = (buffer: AudioBuffer, rate = 1) => { + const s = ctx.createBufferSource(); + s.buffer = buffer; + s.loop = true; + s.playbackRate.value = rate; + return s; + }; + const filter = (type: BiquadFilterType, freq: number, q = 0.7) => { + const f = ctx.createBiquadFilter(); + f.type = type; + f.frequency.value = freq; + f.Q.value = q; + return f; + }; + const gain = (v = 0) => { + const n = ctx.createGain(); + n.gain.value = v; + return n; + }; + // Two decorrelated pink loops (different rates and offsets) and one brown. + const pinkA = loop(buffers.pink); + const pinkB = loop(buffers.pink, 0.93); + const brown = loop(buffers.brown); + + // Breeze: band-passed pink, swept slowly, two loops panned ±0.6, softened by a 600 Hz lowpass. + const bpL = filter('bandpass', 560, 0.9); + const bpR = filter('bandpass', 600, 0.9); + const panL = createPan(ctx, -0.6); + const panR = createPan(ctx, 0.6); + const lp = filter('lowpass', 600, 0.3); + this.breezeGain = gain(); + pinkA.connect(bpL).connect(panL.node).connect(lp); + pinkB.connect(bpR).connect(panR.node).connect(lp); + lp.connect(this.breezeGain).connect(g.outdoor); + this.lfo = ctx.createOscillator(); + this.lfo.frequency.value = 0.08; + const depth = gain(320); + this.lfo.connect(depth); + depth.connect(bpL.frequency); + depth.connect(bpR.frequency); + + // High wind: pink above 1.2 kHz, in gusts. + const hp = filter('highpass', 1200, 0.5); + this.windGain = gain(); + pinkA.connect(hp).connect(this.windGain).connect(g.outdoor); + + // Crickets: one sine, two alternating voices (pitch and side), pulsed by an envelope. + this.cricketOsc = ctx.createOscillator(); + this.cricketOsc.frequency.value = 4400; + this.cricketEnv = gain(); + const cp = createPan(ctx, 0); + this.cricketPan = cp.pan; + this.cricketOsc.connect(this.cricketEnv).connect(cp.node).connect(g.outdoor); + + // Murmur: pink through two speech-ish bands, gated into syllables, panned to its source. + const m1 = filter('bandpass', 450, 0.8); + const m2 = filter('bandpass', 1100, 1.2); + this.murmurGain = gain(); + const mp = createPan(ctx, 0); + this.murmurPan = mp.pan; + pinkB.connect(m1).connect(this.murmurGain); + pinkB.connect(m2).connect(this.murmurGain); + this.murmurGain.connect(mp.node).connect(g.outdoor); + + // Water: a narrow 1.8 kHz band in short grains, plus a low rumble. + const wbp = filter('bandpass', 1800, 3); + const wlp = filter('lowpass', 400, 0.5); + this.waterGain = gain(); + this.rumble = gain(); + const wp = createPan(ctx, 0); + this.waterPan = wp.pan; + pinkA.connect(wbp).connect(this.waterGain); + pinkA.connect(wlp).connect(this.rumble).connect(this.waterGain); + this.waterGain.connect(wp.node).connect(g.outdoor); + + // Traffic: brown noise under 140 Hz; a pass-by sweeps it 200 → 120 Hz. + this.trafficLp = filter('lowpass', 140, 0.6); + this.trafficGain = gain(); + brown.connect(this.trafficLp).connect(this.trafficGain).connect(g.outdoor); + + // Room tone: brown noise; inside, the outdoor lowpass (700 Hz) shapes it. + this.roomGain = gain(); + brown.connect(this.roomGain).connect(g.outdoor); + + this.sources = [pinkA, pinkB, brown, this.lfo, this.cricketOsc]; + } + + start(t: number): void { + const [pinkA, pinkB, brown, lfo, crickets] = this.sources as [AudioBufferSourceNode, AudioBufferSourceNode, AudioBufferSourceNode, OscillatorNode, OscillatorNode]; + pinkA.start(t, 0); + pinkB.start(t, 1.7); + brown.start(t, 0.9); + lfo.start(t); + crickets.start(t); + for (const k of Object.keys(this.next) as (keyof typeof this.next)[]) this.next[k] = t; + this.next.pass = t + between(this.rng, 4, 10); + } + + stop(t: number): void { + for (const s of this.sources) { + try { + s.stop(t); + } catch { + // never started + } + } + } + + /** Follow the mix at audio time `now` (dt = seconds since the last update) and schedule the next LOOKAHEAD. */ + update(now: number, mix: AmbientMix, dt: number): void { + const { rng, L } = this; + const step = Math.min(1, Math.max(0, dt)); + for (const k of LEVEL_KEYS) L[k] = smoothToward(L[k], mix[k], step, LEVEL_TAU); + L.day = mix.day; + this.pans.murmur = smoothToward(this.pans.murmur, mix.murmurPan, step, 0.3); + this.pans.water = smoothToward(this.pans.water, mix.waterPan, step, 0.3); + const horizon = now + LOOKAHEAD; + + // Continuous layers: glide toward their level. + this.breezeGain.gain.setTargetAtTime(L.breeze * TRIM.breeze, now, 0.15); + if (now >= this.next.gust) { + this.gust = between(rng, 0.35, 1); + this.next.gust = now + between(rng, 3, 6); + } + this.windGain.gain.setTargetAtTime(L.wind * TRIM.wind * this.gust, now, 1.2); + this.roomGain.gain.setTargetAtTime(L.room * TRIM.room, now, 0.3); + this.rumble.gain.setTargetAtTime(TRIM.rumble, now, 0.3); + this.murmurPan?.setTargetAtTime(this.pans.murmur, now, 0.3); + this.waterPan?.setTargetAtTime(this.pans.water, now, 0.3); + if (now >= this.next.lfo) { + this.lfo.frequency.setTargetAtTime(between(rng, 0.05, 0.12), now, 4); + this.next.lfo = now + between(rng, 15, 25); + } + + // Inside a building: the outdoor bed −9 dB through 700 Hz. + if (mix.interior !== this.inside) { + this.inside = mix.interior; + this.g.outdoor.gain.setTargetAtTime(this.inside ? INTERIOR_OUTDOOR_GAIN : 1, now, 0.25); + this.g.outdoorLp.frequency.setTargetAtTime(this.inside ? OUTDOOR_INSIDE_HZ : OUTDOOR_OPEN_HZ, now, 0.25); + } + + // Traffic: a steady hum, and now and then (by day) a distant car passing. + if (!this.pass && now >= this.next.pass) { + if (rng() < L.day && L.traffic > 0.01) this.pass = { start: now + 0.05, dur: 2.5 }; + this.next.pass = now + between(rng, 8, 20); + } + let swell = 0; + let freq = 140; + if (this.pass) { + const u = (now - this.pass.start) / this.pass.dur; + if (u >= 1) this.pass = null; + else if (u >= 0) { + swell = Math.sin(Math.PI * u) ** 2; + freq = 200 - 80 * u; + } + } + this.trafficGain.gain.setTargetAtTime(L.traffic * TRIM.traffic * (1 + 1.5 * swell), now, 0.3); + this.trafficLp.frequency.setTargetAtTime(freq, now, 0.3); + + // Murmur syllables. + const murmur = L.murmur * TRIM.murmur; + let t = Math.max(this.next.syllable, now); + if (murmur > 1e-4 || this.murmurOn) { + while (t < horizon) { + this.murmurGain.gain.setTargetAtTime(murmur > 1e-4 ? murmur * between(rng, 0.4, 1) : 0, t, 0.035); + t += between(rng, 0.12, 0.3); + } + this.murmurOn = murmur > 1e-4; + } else t = horizon; + this.next.syllable = t; + + // Water grains. + const water = L.water * TRIM.water; + t = Math.max(this.next.grain, now); + if (water > 1e-4 || this.waterOn) { + while (t < horizon) { + const d = between(rng, 0.04, 0.09); + this.waterGain.gain.setTargetAtTime(water * between(rng, 0.6, 1), t, 0.01); + this.waterGain.gain.setTargetAtTime(water * 0.3, t + d * 0.5, d * 0.4); + t += d + between(rng, 0, 0.07); + } + this.waterOn = water > 1e-4; + if (!this.waterOn) this.waterGain.gain.setTargetAtTime(0, t, 0.1); + } else t = horizon; + this.next.grain = t; + + // Cricket trains: 3–5 pulses of 18 ms at 30 Hz, voices alternating. + t = Math.max(this.next.train, now); + while (t < horizon) { + const level = L.crickets * TRIM.crickets; + if (level < 1e-4) { + t += 0.4; + continue; + } + this.cricketVoice ^= 1; + const a = this.cricketVoice === 1; + this.cricketOsc.frequency.setValueAtTime(a ? 4400 + between(rng, -150, 150) : 4250 + between(rng, -100, 100), t); + this.cricketPan?.setValueAtTime(a ? -0.4 : 0.35, t); + const n = 3 + Math.floor(rng() * 3); + for (let i = 0; i < n; i++) { + const tp = t + i / 30; + this.cricketEnv.gain.setValueAtTime(0, tp); + this.cricketEnv.gain.linearRampToValueAtTime(level, tp + 0.004); + this.cricketEnv.gain.linearRampToValueAtTime(0, tp + 0.018); + } + t += n / 30 + between(rng, 0.2, 0.6); + } + this.next.train = t; + + // Bird calls: a Poisson process at birdsRate calls/s. + const rate = L.birdsRate; + if (rate < 0.005) { + this.birdsOn = false; + } else { + const wait = () => Math.min(30, Math.max(0.25, -Math.log(1 - rng()) / rate)); + if (!this.birdsOn) this.next.call = now + wait(); + this.birdsOn = true; + while (this.next.call < horizon) { + this.spawnBird(Math.max(now, this.next.call), Math.floor(rng() * 3), between(rng, -0.7, 0.7), TRIM.birds * between(rng, 0.5, 1)); + this.next.call += wait(); + } + } + } + + get levels(): BedLevels { + const r = (v: number) => Math.round(v * 1000) / 1000; + const { L } = this; + return { + breeze: r(L.breeze), + wind: r(L.wind * this.gust), + birds: r(L.birdsRate), + crickets: r(L.crickets), + murmur: r(L.murmur), + water: r(L.water), + traffic: r(L.traffic), + room: r(L.room), + interior: this.inside, + passing: this.pass !== null, + }; + } +} diff --git a/apps/studio/src/audio/audio.module.css b/apps/studio/src/audio/audio.module.css new file mode 100644 index 0000000..d80710e --- /dev/null +++ b/apps/studio/src/audio/audio.module.css @@ -0,0 +1,143 @@ +/* Sound in the HUD: the speaker toggle (inside the glass clock widget) and the Esc menu's Sound tab. */ + +.toggle { + display: inline-flex; + align-items: center; + gap: 6px; + height: 36px; + min-width: 36px; + padding: 0 10px 0 8px; + border-radius: 8px; + color: var(--rl-text); + font-size: var(--rl-text-xs); + font-weight: 600; + white-space: nowrap; +} +.toggle[data-compact] { + justify-content: center; + padding: 0; +} +.toggle:hover:not(:disabled) { + background: var(--rl-surface-2); +} +.toggle:focus-visible { + outline: none; + box-shadow: var(--rl-focus-ring); +} +.toggle[aria-pressed='false'] { + color: var(--rl-text-muted); +} +.toggle:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.settings { + display: grid; +} +.status { + margin-bottom: 4px; + color: var(--rl-text-muted); + font-size: var(--rl-text-xs); +} +.row { + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + align-items: center; + gap: 16px; + padding: 12px 0; + box-shadow: inset 0 -1px 0 var(--rl-line); +} +.label { + display: block; + color: var(--rl-text); + font-weight: 600; +} +.hint, +.note { + color: var(--rl-text-muted); + font-size: var(--rl-text-xs); + line-height: 1.5; +} +.note { + margin-top: 12px; +} +.control { + display: flex; + align-items: center; + justify-content: flex-end; + gap: 8px; +} +.sliderControl { + display: grid; + grid-template-columns: 168px 40px 80px; + align-items: center; + gap: 8px; +} +.value { + min-width: 36px; + color: var(--rl-text-muted); + font-size: var(--rl-text-xs); + font-variant-numeric: tabular-nums; + text-align: right; +} + +.range { + width: 168px; + height: 20px; + margin: 0; + background: transparent; + appearance: none; + cursor: pointer; +} +.range:disabled { + opacity: 0.5; + cursor: not-allowed; +} +.range::-webkit-slider-runnable-track { + height: 4px; + border-radius: 999px; + background: linear-gradient(to right, var(--rl-accent-fill) var(--fill, 50%), var(--rl-surface-3) var(--fill, 50%)); +} +.range::-moz-range-track { + height: 4px; + border-radius: 999px; + background: var(--rl-surface-3); +} +.range::-moz-range-progress { + height: 4px; + border-radius: 999px; + background: var(--rl-accent-fill); +} +.range::-webkit-slider-thumb { + width: 16px; + height: 16px; + margin-top: -6px; + border: 2px solid var(--rl-surface); + border-radius: 50%; + background: var(--rl-text); + appearance: none; +} +.range::-moz-range-thumb { + width: 12px; + height: 12px; + border: 2px solid var(--rl-surface); + border-radius: 50%; + background: var(--rl-text); +} + +@media (max-width: 560px) { + .row { + grid-template-columns: 1fr; + gap: 8px; + } + .control { + justify-content: flex-start; + } + .sliderControl { + grid-template-columns: minmax(0, 1fr) 40px 80px; + } + .range { + width: 100%; + } +} diff --git a/apps/studio/src/audio/check/index.html b/apps/studio/src/audio/check/index.html new file mode 100644 index 0000000..5b89d80 --- /dev/null +++ b/apps/studio/src/audio/check/index.html @@ -0,0 +1,122 @@ + + + + + + Runloop sound board + + + +
+

Runloop sound board

+

+ Every sound in Studio, synthesized live with the Web Audio API (the same code as the town). Sound starts with your first click. Event cue + buttons are previews: in the town they play only for real agent events. +

+
+
+ + + diff --git a/apps/studio/src/audio/check/main.ts b/apps/studio/src/audio/check/main.ts new file mode 100644 index 0000000..d051dc6 --- /dev/null +++ b/apps/studio/src/audio/check/main.ts @@ -0,0 +1,146 @@ +/** + * The sound board (`node apps/studio/src/audio/check/render-check.mjs --serve`, + * port 3074): every sound live through the real engine, an ambient bed driven + * by a made-up listener, the debug hook, and the offline render check + * (`?check=1` runs it on load and publishes `window.__renderCheck`). + */ +import { getAudioEngine } from '../engine'; +import { SCENES, renderLayers, runRenderCheck, type CheckRow, type Measure } from '../render'; +import { EVENT_CUES, UI_SOUNDS, type AudioListener, type AudioSettings } from '../types'; + +declare global { + interface Window { + __renderCheck?: { ok: boolean; rows: CheckRow[] } | { error: string }; + __renderLayers?: Measure[]; + } +} + +const engine = getAudioEngine()!; +const app = document.getElementById('app')!; + +function el(tag: K, props: Partial = {}, ...children: (Node | string)[]): HTMLElementTagNameMap[K] { + const e = Object.assign(document.createElement(tag), props); + e.append(...children); + return e; +} +const button = (text: string, onClick: () => void) => el('button', { type: 'button', onclick: onClick }, text); +const section = (title: string, ...children: Node[]) => el('section', {}, el('h2', {}, title), el('div', { className: 'panel' }, ...children)); + +function slider(label: string, value: number, onInput: (v: number) => void): HTMLLabelElement { + const out = el('output', {}, `${Math.round(value * 100)}%`); + const input = el('input', { type: 'range', min: '0', max: '100', value: String(Math.round(value * 100)) }); + input.oninput = () => { + out.textContent = `${input.value}%`; + onInput(Number(input.value) / 100); + }; + return el('label', {}, label, input, out); +} + +// ── Volumes ── +const s = engine.getSettings(); +const vol = (k: keyof Pick, label: string) => slider(label, s[k], (v) => engine.update({ [k]: v })); +const flag = (k: 'enabled' | 'reduce', label: string) => { + const box = el('input', { type: 'checkbox', checked: s[k] }); + box.onchange = () => { + engine.update({ [k]: box.checked }); + engine.unlock(); + }; + return el('label', {}, label, box, ''); +}; + +// ── Ambient bed: a made-up listener, as the town would report it (~4 Hz) ── +let scene = Object.keys(SCENES)[1]!; +const bedState = { on: false, zoom01: 1, night: 0, paused: false }; +const sceneSelect = el('select', {}, ...Object.keys(SCENES).map((name) => el('option', { value: name, selected: name === scene }, name))); +sceneSelect.onchange = () => { + scene = sceneSelect.value; + const l = SCENES[scene]!; + bedState.zoom01 = l.zoom01; + bedState.night = l.night; +}; +const bedButton = button('Start the bed', () => { + bedState.on = !bedState.on; + bedButton.textContent = bedState.on ? 'Stop the bed' : 'Start the bed'; + engine.unlock(); +}); +setInterval(() => { + if (!bedState.on) return; + const l: AudioListener = { ...SCENES[scene]!, zoom01: bedState.zoom01, night: bedState.night, paused: bedState.paused }; + engine.setListener(l); +}, 250); +const pausedBox = el('input', { type: 'checkbox' }); +pausedBox.onchange = () => (bedState.paused = pausedBox.checked); + +// ── Check results ── +const results = el('div'); +function showResults(r: { ok: boolean; rows: CheckRow[] }) { + const f = (v: number | undefined) => (v === undefined || !Number.isFinite(v) ? '−∞' : v.toFixed(1)); + results.replaceChildren( + el('p', { className: r.ok ? 'ok' : 'bad' }, r.ok ? `All ${r.rows.length} checks pass.` : `${r.rows.filter((x) => !x.ok).length} of ${r.rows.length} checks fail.`), + el( + 'table', + {}, + el('tr', {}, el('th', {}, ''), el('th', {}, 'Sound'), el('th', {}, 'Peak dBFS'), el('th', {}, 'RMS dBFS'), el('th', {}, 'Expect')), + ...r.rows.map((row) => + el( + 'tr', + {}, + el('td', { className: row.ok ? 'ok' : 'bad' }, row.ok ? '✓ pass' : '✗ fail'), + el('td', {}, `${row.group} · ${row.name}`), + el('td', {}, f(row.measure?.peakDb)), + el('td', {}, f(row.measure?.rmsDb)), + el('td', {}, row.expect), + ), + ), + ), + ); +} +async function check() { + results.replaceChildren(el('p', {}, 'Rendering offline…')); + try { + const r = await runRenderCheck(); + window.__renderCheck = r; + showResults(r); + } catch (e) { + window.__renderCheck = { error: e instanceof Error ? `${e.message}\n${e.stack ?? ''}` : String(e) }; + results.replaceChildren(el('p', { className: 'bad' }, `Render check failed to run: ${String(e)}`)); + } +} + +const debug = el('pre'); +setInterval(() => { + debug.textContent = JSON.stringify((window as unknown as { __runloopAudio: unknown }).__runloopAudio, null, 1); +}, 500); + +app.append( + section('Volumes', flag('enabled', 'Sound on'), vol('master', 'Master'), vol('ambient', 'Ambient'), vol('ui', 'Interface'), vol('events', 'Agent events'), flag('reduce', 'Reduce sound')), + section( + 'Ambient bed', + el('div', { className: 'row' }, bedButton, sceneSelect), + slider('Zoom (town → street)', bedState.zoom01, (v) => (bedState.zoom01 = v)), + slider('Night', bedState.night, (v) => (bedState.night = v)), + el('label', {}, 'World paused', pausedBox, ''), + ), + section( + 'Interface sounds', + el('div', { className: 'row' }, ...UI_SOUNDS.map((u) => button(u, () => engine.ui(u))), button('whoosh (fly, pan →)', () => engine.whoosh(700, 0.6))), + ), + section('Event cue previews (real agent events only, in the town)', el('div', { className: 'row' }, ...EVENT_CUES.map((c) => button(c, () => engine.previewCue(c))))), + section('Offline render check', el('div', { className: 'row' }, button('Run the render check', () => void check())), results), + section('window.__runloopAudio', debug), +); + +const q = new URLSearchParams(location.search); +if (q.has('check')) void check(); +if (q.has('ui')) { + const host = el('div', { id: 'ui-preview' }); + app.before(host); + void import('./ui').then((m) => m.mountUiPreview(host)); +} +if (q.has('layers')) { + void (async () => { + const all: Measure[] = []; + for (const name of Object.keys(SCENES)) all.push(...(await renderLayers(name))); + window.__renderLayers = all; + })(); +} diff --git a/apps/studio/src/audio/check/next-link-stub.tsx b/apps/studio/src/audio/check/next-link-stub.tsx new file mode 100644 index 0000000..4ce4bc3 --- /dev/null +++ b/apps/studio/src/audio/check/next-link-stub.tsx @@ -0,0 +1,7 @@ +/** The sound board runs outside Next: `next/link` is a plain anchor here (aliased in render-check.mjs). */ +import { forwardRef, type AnchorHTMLAttributes } from 'react'; + +const Link = forwardRef & { href: string }>(function Link(props, ref) { + return ; +}); +export default Link; diff --git a/apps/studio/src/audio/check/render-check.mjs b/apps/studio/src/audio/check/render-check.mjs new file mode 100644 index 0000000..f11f15f --- /dev/null +++ b/apps/studio/src/audio/check/render-check.mjs @@ -0,0 +1,106 @@ +#!/usr/bin/env node +// The soundscape's browser check (contract §4b, S1 and S4). Nobody can listen in CI, so this measures: +// 1. live, in headless Chromium with the real autoplay policy: no AudioContext before the first gesture, +// a click starts it, the bed follows a listener, a hidden tab suspends it and showing it resumes; +// 2. offline (OfflineAudioContext): every UI sound, the whoosh and the four cues are non-silent and never +// clip (peak < 0.9) at default and full volume; the ambient bed is quiet (RMS bounds); the busiest +// mix at full volume does not clip; every setting (master, ambient, UI, events, mute, reduce) changes +// the output. +// +// node apps/studio/src/audio/check/render-check.mjs # run the checks (exit 1 on failure) +// node apps/studio/src/audio/check/render-check.mjs --serve # keep the sound board up (port 3074) to listen; +// # ?ui=1 shows the HUD toggle and the Sound settings +// [--port 3074] [--json out.json] +import { createServer } from 'vite'; +import { chromium } from 'playwright'; +import fs from 'node:fs'; +import path from 'node:path'; + +const HERE = import.meta.dirname; +const ROOT = path.resolve(HERE, '../../../../..'); +const args = process.argv.slice(2); +const opt = (name, def) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : def; +}; +const PORT = Number(opt('--port', '3074')); +const SERVE = args.includes('--serve'); +const JSON_OUT = opt('--json', null); + +const server = await createServer({ + configFile: false, + root: HERE, + logLevel: 'warn', + server: { port: PORT, strictPort: true, host: '127.0.0.1', fs: { allow: [ROOT] } }, + // `?ui=1` mounts the React toggle and settings outside Next. + resolve: { alias: { 'next/link': path.join(HERE, 'next-link-stub.tsx') } }, +}); +await server.listen(); +const url = `http://127.0.0.1:${PORT}/`; +if (SERVE) { + console.log(`Sound board: ${url} (offline check: ${url}?check=1)`); +} else { + let failed = 0; + const line = (ok, text) => { + if (!ok) failed++; + console.log(`${ok ? 'PASS' : 'FAIL'} ${text}`); + }; + const browser = await chromium.launch({ args: ['--autoplay-policy=user-gesture-required'] }); + try { + const page = await browser.newPage(); + const errors = []; + page.on('pageerror', (e) => errors.push(e.message)); + page.on('console', (m) => m.type() === 'error' && errors.push(m.text())); + const audio = () => page.evaluate(() => window.__runloopAudio); + + // ── Live (S1) ── + await page.goto(url, { waitUntil: 'load' }); + await page.waitForTimeout(800); + let a = await audio(); + line(a.created === false && a.state === 'locked', `no AudioContext before the first gesture (state ${a.state}, created ${a.created})`); + await page.getByRole('button', { name: 'Start the bed' }).click(); + await page.waitForTimeout(300); + a = await audio(); + line(a.created === true && a.state === 'running', `the first click starts sound (state ${a.state})`); + await page.waitForTimeout(3000); + a = await audio(); + line(a.active.includes('breeze') && a.active.includes('murmur') && a.buses.ambient > 0.2, `the bed fades in and follows the listener (active: ${a.active.join(', ')}; ambient bus ${a.buses.ambient})`); + line(a.persistentNodes <= 40, `persistent nodes ${a.persistentNodes} ≤ 40`); + await page.evaluate(() => { + Object.defineProperty(document, 'visibilityState', { configurable: true, get: () => 'hidden' }); + document.dispatchEvent(new Event('visibilitychange')); + }); + await page.waitForTimeout(500); + a = await audio(); + line(a.state === 'suspended' && a.buses.master === 0, `a hidden tab mutes and suspends (state ${a.state}, master ${a.buses.master})`); + await page.evaluate(() => { + Object.defineProperty(document, 'visibilityState', { configurable: true, get: () => 'visible' }); + document.dispatchEvent(new Event('visibilitychange')); + }); + await page.waitForTimeout(800); + a = await audio(); + line(a.state === 'running' && a.buses.master > 0.28, `showing it resumes and fades back in (state ${a.state}, master ${a.buses.master})`); + await page.getByRole('button', { name: 'needsApproval' }).click(); + await page.getByRole('button', { name: 'select' }).click(); + await page.waitForTimeout(100); + a = await audio(); + line(a.voices >= 1 && a.voices <= 12, `one-shot voices play within the cap (${a.voices})`); + + // ── Offline (S4) ── + await page.goto(`${url}?check=1`, { waitUntil: 'load' }); + await page.waitForFunction(() => window.__renderCheck, null, { timeout: 180_000 }); + const r = await page.evaluate(() => window.__renderCheck); + if (r.error) line(false, `render check crashed: ${r.error}`); + else { + const f = (v) => (v === null || !Number.isFinite(v) ? ' −∞ ' : v.toFixed(1).padStart(5)); + for (const row of r.rows) line(row.ok, `${row.group.padEnd(8)} ${row.name.padEnd(36)} peak ${f(row.measure?.peakDb)} dBFS rms ${f(row.measure?.rmsDb)} dBFS (${row.expect})`); + if (JSON_OUT) fs.writeFileSync(JSON_OUT, JSON.stringify(r, null, 1)); + } + line(errors.length === 0, `no page errors${errors.length ? `: ${errors.slice(0, 3).join(' | ')}` : ''}`); + } finally { + await browser.close(); + await server.close(); + } + console.log(failed ? `\n${failed} check(s) failed.` : '\nAll sound checks passed.'); + process.exit(failed ? 1 : 0); +} diff --git a/apps/studio/src/audio/check/ui.tsx b/apps/studio/src/audio/check/ui.tsx new file mode 100644 index 0000000..4cd9cfb --- /dev/null +++ b/apps/studio/src/audio/check/ui.tsx @@ -0,0 +1,51 @@ +/** + * `?ui=1` on the sound board: the HUD speaker toggle and the Esc menu's Sound + * tab, mounted alone over a daytime-sky backdrop (to review them before the + * HUD wires them in). `&theme=light` for the light theme. + */ +import type { CSSProperties } from 'react'; +import { createRoot } from 'react-dom/client'; +import '../../app/tokens.css'; +import '../../app/globals.css'; +import { SoundSettings, SoundToggle } from '../react'; + +const glass: CSSProperties = { + display: 'inline-flex', + alignItems: 'center', + gap: 10, + padding: '6px 8px 6px 12px', + borderRadius: 12, + background: 'var(--rl-glass)', + border: '1px solid var(--rl-glass-line)', + backdropFilter: 'blur(14px) saturate(1.2)', + color: 'var(--rl-text-muted)', + boxShadow: 'var(--rl-shadow-pop)', +}; + +function Preview() { + return ( +
+
+ Day 2 · 14:20 + ⏸ 1× 4× 16× +
+
+ +
+
+

Sound

+ +
+
+ ); +} + +export function mountUiPreview(into: HTMLElement): void { + const theme = new URLSearchParams(location.search).get('theme'); + if (theme) document.documentElement.dataset.theme = theme; + // Next sets the Geist variables; here, the system font stands in. + document.documentElement.style.setProperty('--font-geist-sans', 'Geist, ui-sans-serif, system-ui, sans-serif'); + createRoot(into).render(); +} diff --git a/apps/studio/src/audio/cues.test.ts b/apps/studio/src/audio/cues.test.ts new file mode 100644 index 0000000..5fb14d2 --- /dev/null +++ b/apps/studio/src/audio/cues.test.ts @@ -0,0 +1,251 @@ +import { WORLD_EVENT_PROV, WORLD_EVENT_TYPES, type WorldEvent, type WorldProvenance } from '@runloop-world/events'; +import { describe, expect, it } from 'vitest'; +import { fixtureEvents } from '../world/test-utils'; +import { CUE_LIMITS, CueLimiter, HOUSED_AGENT, classifyCue, cueAgentId, nextAgentState } from './cues'; +import type { AgentAudioState, EventCue } from './types'; + +const AGENT = 'agt_00000000-0000-4000-8000-000000000001'; +const OTHER = 'agt_00000000-0000-4000-8000-000000000002'; +const RUN = '00000000-0000-4000-8000-0000000000aa'; + +let seq = 100; +function ev(type: string, p: Record, prov: WorldProvenance = 'system'): WorldEvent { + return { seq: ++seq, tick: 1000 + seq, type, prov, p, actor: AGENT } as unknown as WorldEvent; +} +const conn = (health: string, attention = 'none') => ev('agent.connection_changed', { agentId: AGENT, health, attention }); +const state = (p: Partial = {}): AgentAudioState => ({ ...HOUSED_AGENT, ...p }); + +/** A payload per event type that would cue if anything could (agent types) or a plain one. */ +function samplePayload(type: string): Record { + switch (type) { + case 'agent.connection_changed': + return { agentId: AGENT, health: 'live', attention: 'permission_requested' }; + case 'agent.activity_observed': + return { agentId: AGENT, runId: RUN, counts: {}, execState: 'working', helpers: 0, openIncidents: 3 }; + case 'agent.dispatch_changed': + return { agentId: AGENT, dispatchId: 'dsp_1', phase: 'finished', outcome: 'failed' }; + case 'agent.session_changed': + return { agentId: AGENT, runId: RUN, phase: 'ended', outcome: 'succeeded' }; + case 'agent.housed': + return { agentId: AGENT, lotId: 'lot_1', buildingId: 'bld_1', displayName: 'Engineering House', hue: '#5aa9f5' }; + default: + return { r: 1, a: 1, b: 2, buildingId: 'bld_corner_cafe', open: true, speed: 4, prevSpeed: 1 }; + } +} + +describe('classifyCue: truth rule (S2)', () => { + it('never cues a simulation event, for every event type', () => { + for (const type of WORLD_EVENT_TYPES) { + expect(classifyCue(ev(type, samplePayload(type), 'simulation'), state()), type).toBeNull(); + } + }); + + it('never cues a non-agent event, whatever its provenance', () => { + const provs: WorldProvenance[] = ['simulation', 'user', 'external-agent', 'system', 'inferred']; + for (const type of WORLD_EVENT_TYPES.filter((t) => !t.startsWith('agent.'))) { + for (const prov of provs) expect(classifyCue(ev(type, samplePayload(type), prov), state()), `${type}/${prov}`).toBeNull(); + } + }); + + it('cues the agent types only with the provenances the schema allows them', () => { + for (const type of WORLD_EVENT_TYPES.filter((t) => t.startsWith('agent.') && t !== 'agent.housed')) { + for (const prov of WORLD_EVENT_PROV[type]) expect(classifyCue(ev(type, samplePayload(type), prov), state()), `${type}/${prov}`).not.toBeNull(); + } + expect(classifyCue(ev('agent.housed', samplePayload('agent.housed'), 'user'), undefined)).toBeNull(); + }); + + it('a recorded engine day (residents, interactions, the clock) makes no cue at all', () => { + const events = fixtureEvents(); + expect(events.length).toBeGreaterThan(100); + const states = new Map(); + let simulated = 0; + for (const e of events) { + const id = cueAgentId(e); + const cue = classifyCue(e, id ? states.get(id) : undefined); + if (e.prov === 'simulation') { + simulated++; + expect(cue).toBeNull(); + } + if (!e.type.startsWith('agent.')) expect(cue).toBeNull(); + if (id) { + const next = nextAgentState(states.get(id), e); + if (next) states.set(id, next); + } + } + expect(simulated).toBeGreaterThan(100); + }); +}); + +describe('classifyCue: the four cues', () => { + it('agentLive when health becomes live (from any other health)', () => { + expect(classifyCue(conn('live'), state({ health: 'awaiting_first_event' }))).toBe('agentLive'); + expect(classifyCue(conn('live'), state({ health: 'idle' }))).toBe('agentLive'); + expect(classifyCue(conn('live'), undefined)).toBe('agentLive'); + expect(classifyCue(conn('live'), state({ health: 'live' }))).toBeNull(); + expect(classifyCue(conn('idle'), state({ health: 'live' }))).toBeNull(); + expect(classifyCue(conn('offline'), state({ health: 'live' }))).toBeNull(); + }); + + it('needsApproval when attention becomes a permission or dispatch wait; it outranks agentLive', () => { + expect(classifyCue(conn('live', 'permission_requested'), state({ health: 'live' }))).toBe('needsApproval'); + expect(classifyCue(conn('idle', 'dispatch_awaiting_approval'), state({ health: 'idle' }))).toBe('needsApproval'); + expect(classifyCue(conn('live', 'permission_requested'), state({ health: 'idle' }))).toBe('needsApproval'); + // Already waiting: no new cue. + expect(classifyCue(conn('live', 'permission_requested'), state({ health: 'live', attention: 'permission_requested' }))).toBeNull(); + // Resolved: no cue. + expect(classifyCue(conn('live', 'none'), state({ health: 'live', attention: 'permission_requested' }))).toBeNull(); + }); + + it('incident when attention becomes incident or open incidents rise', () => { + expect(classifyCue(conn('live', 'incident'), state({ health: 'live' }))).toBe('incident'); + const act = (n: number, prov: WorldProvenance = 'external-agent') => + ev('agent.activity_observed', { agentId: AGENT, runId: RUN, counts: {}, execState: 'working', helpers: 0, openIncidents: n }, prov); + expect(classifyCue(act(1), state({ openIncidents: 0 }))).toBe('incident'); + expect(classifyCue(act(2, 'inferred'), state({ openIncidents: 1 }))).toBe('incident'); + expect(classifyCue(act(1), state({ openIncidents: 1 }))).toBeNull(); + expect(classifyCue(act(0), state({ openIncidents: 2 }))).toBeNull(); + }); + + it('dispatch outcomes: completed → taskDone; failures → incident; cancelled and unfinished phases → none', () => { + const d = (phase: string, outcome?: string) => ev('agent.dispatch_changed', { agentId: AGENT, dispatchId: 'dsp_1', phase, ...(outcome ? { outcome } : {}) }, 'user'); + const table: [string, string | undefined, EventCue | null][] = [ + ['finished', 'completed', 'taskDone'], + ['finished', 'failed', 'incident'], + ['finished', 'timed_out', 'incident'], + ['finished', 'budget_exceeded', 'incident'], + ['finished', 'lost', 'incident'], + ['finished', 'cancelled', null], + ['approved', undefined, null], + ['running', undefined, null], + ]; + for (const [phase, outcome, cue] of table) expect(classifyCue(d(phase, outcome), state()), `${phase}/${outcome}`).toBe(cue); + }); + + it('session ends: succeeded → taskDone; failed → incident; stopped, interrupted and starts → none', () => { + const s = (phase: string, outcome?: string) => ev('agent.session_changed', { agentId: AGENT, runId: RUN, phase, ...(outcome ? { outcome } : {}) }, 'external-agent'); + expect(classifyCue(s('ended', 'succeeded'), state())).toBe('taskDone'); + expect(classifyCue(s('ended', 'failed'), state())).toBe('incident'); + expect(classifyCue(s('ended', 'stopped'), state())).toBeNull(); + expect(classifyCue(s('ended', 'interrupted'), state())).toBeNull(); + expect(classifyCue(s('started'), state())).toBeNull(); + }); + + it('folds agent state the way applyEvent does', () => { + let st = nextAgentState(undefined, ev('agent.housed', samplePayload('agent.housed'), 'user')); + expect(st).toEqual(HOUSED_AGENT); + st = nextAgentState(st, conn('live', 'incident')); + expect(st).toMatchObject({ health: 'live', attention: 'incident' }); + st = nextAgentState(st, ev('agent.activity_observed', { agentId: AGENT, runId: RUN, counts: {}, execState: 'working', helpers: 0, openIncidents: 2 }, 'external-agent')); + expect(st?.openIncidents).toBe(2); + st = nextAgentState(st, ev('agent.session_changed', { agentId: AGENT, runId: RUN, phase: 'started' }, 'external-agent')); + expect(st?.openIncidents).toBe(0); + expect(nextAgentState(st, ev('resident.arrived', {}, 'simulation'))).toBe(st); + }); +}); + +describe('CueLimiter (S3)', () => { + const setup = () => { + let t = 0; + const clock = { now: () => t, at: (ms: number) => (t = ms), add: (ms: number) => (t += ms) }; + return { clock, lim: new CueLimiter(clock.now) }; + }; + + it('spaces cues 1.2 s apart through a short queue', () => { + const { clock, lim } = setup(); + expect(lim.offer('agentLive', AGENT)).toBe('play'); + expect(lim.offer('taskDone', OTHER)).toBe('queued'); + clock.at(1000); + expect(lim.due()).toEqual([]); + clock.at(1200); + expect(lim.due().map((c) => c.cue)).toEqual(['taskDone']); + expect(lim.due()).toEqual([]); + clock.at(2500); + expect(lim.offer('incident', OTHER)).toBe('play'); + }); + + it('keeps at most 2 waiting, strongest first; a weaker cue cannot push out a stronger one', () => { + const { clock, lim } = setup(); + expect(lim.offer('agentLive', AGENT)).toBe('play'); + expect(lim.offer('taskDone', AGENT)).toBe('queued'); + expect(lim.offer('agentLive', OTHER)).toBe('queued'); + expect(lim.offer('agentLive', 'agt_x')).toBe('queue_full'); + expect(lim.offer('needsApproval', OTHER)).toBe('queued'); // pushes out the waiting agentLive + expect(lim.waiting).toBe(2); + clock.at(1200); + expect(lim.due()[0]?.cue).toBe('needsApproval'); + clock.at(2400); + expect(lim.due()[0]?.cue).toBe('taskDone'); + // The pushed-out cue gave its cooldown back. + clock.at(3600); + expect(lim.offer('agentLive', OTHER)).toBe('play'); + }); + + it('drops a waiting cue whose moment has passed', () => { + let t = 0; + const lim = new CueLimiter(() => t, { ...CUE_LIMITS, spacingMs: 10_000 }); + lim.offer('agentLive', AGENT); + expect(lim.offer('taskDone', AGENT)).toBe('queued'); + t = 4500; + expect(lim.due()).toEqual([]); + expect(lim.waiting).toBe(0); + }); + + it('caps cues at 6 per minute', () => { + const { clock, lim } = setup(); + for (let i = 0; i < 6; i++) { + clock.at(i * 1300); + expect(lim.offer('taskDone', `agt_${i}`)).toBe('play'); + } + clock.at(6 * 1300); + expect(lim.offer('needsApproval', 'agt_9')).toBe('rate'); + clock.at(60_001); + expect(lim.offer('needsApproval', 'agt_9')).toBe('play'); + }); + + it('holds each agent+cue for its cooldown (agentLive 60 s, needsApproval 30 s, taskDone 20 s, incident 15 s)', () => { + const cases: [EventCue, number][] = [ + ['agentLive', 60_000], + ['needsApproval', 30_000], + ['taskDone', 20_000], + ['incident', 15_000], + ]; + for (const [cue, ms] of cases) { + const { clock, lim } = setup(); + expect(lim.offer(cue, AGENT)).toBe('play'); + clock.at(ms - 1); + expect(lim.offer(cue, AGENT), cue).toBe('cooldown'); + expect(lim.offer(cue, OTHER), cue).toBe('play'); + clock.at(ms + 5000); + expect(lim.offer(cue, AGENT), cue).toBe('play'); + } + }); + + it('suppresses cues after a resync, and drops what was waiting', () => { + const { clock, lim } = setup(); + lim.offer('agentLive', AGENT); + lim.offer('taskDone', AGENT); + lim.suppressFor(3000); + expect(lim.waiting).toBe(0); + clock.at(2999); + expect(lim.offer('incident', AGENT)).toBe('suppressed'); + expect(lim.due()).toEqual([]); + clock.at(3000); + expect(lim.offer('incident', AGENT)).toBe('play'); + // The dropped taskDone did not keep its cooldown. + clock.at(4300); + expect(lim.offer('taskDone', AGENT)).toBe('play'); + }); + + it('plays nothing while hidden and keeps nothing for later (no burst on return)', () => { + const { clock, lim } = setup(); + lim.offer('agentLive', AGENT); + lim.offer('taskDone', AGENT); + lim.setHidden(true); + expect(lim.waiting).toBe(0); + expect(lim.offer('needsApproval', AGENT)).toBe('hidden'); + clock.at(30_000); + lim.setHidden(false); + expect(lim.due()).toEqual([]); + expect(lim.offer('needsApproval', AGENT)).toBe('play'); + }); +}); diff --git a/apps/studio/src/audio/cues.ts b/apps/studio/src/audio/cues.ts new file mode 100644 index 0000000..51b1df2 --- /dev/null +++ b/apps/studio/src/audio/cues.ts @@ -0,0 +1,209 @@ +/** + * Event cues (contract §4b): which real agent events make a sound, and how + * often. Pure: `classifyCue` looks at one world event and the agent's state + * before it; `CueLimiter` paces what may play, with an injected clock. + * + * Truth rule: only `agent.*` events can cue, never a `simulation` event (the + * sim cannot emit `agent.*` at all, P1; the guard is here anyway). Residents, + * interactions, buildings, the clock and commands never make an event sound. + * A cue never stands alone: the same fact is always a notification with icon + * and text. + */ +import type { WorldEvent } from '@runloop-world/events'; +import type { AgentAudioState, EventCue } from './types'; + +/** A just-housed agent (`agent.housed` folds to this; also the state of an agent the tracker has not seen). */ +export const HOUSED_AGENT: Readonly = Object.freeze({ health: 'not_connected', attention: 'none', openIncidents: 0 }); + +/** Within one event the strongest cue wins. */ +export const CUE_PRIORITY: Readonly> = { needsApproval: 4, incident: 3, taskDone: 2, agentLive: 1 }; + +const strongest = (cues: EventCue[]): EventCue | null => cues.reduce((a, c) => (a && CUE_PRIORITY[a] >= CUE_PRIORITY[c] ? a : c), null); + +/** The agent an `agent.*` event is about (null for any other event). */ +export function cueAgentId(e: WorldEvent): string | null { + if (!e.type.startsWith('agent.')) return null; + const p = e.p as { agentId?: unknown }; + return typeof p.agentId === 'string' ? p.agentId : null; +} + +/** The cue a live world event earns, given the agent's state before it; null for most events. */ +export function classifyCue(e: WorldEvent, prev: AgentAudioState | undefined): EventCue | null { + if (e.prov === 'simulation' || !e.type.startsWith('agent.')) return null; + const before = prev ?? HOUSED_AGENT; + switch (e.type) { + case 'agent.connection_changed': { + const cues: EventCue[] = []; + const { health, attention } = e.p; + if (attention !== before.attention) { + if (attention === 'permission_requested' || attention === 'dispatch_awaiting_approval') cues.push('needsApproval'); + else if (attention === 'incident') cues.push('incident'); + } + if (health === 'live' && before.health !== 'live') cues.push('agentLive'); + return strongest(cues); + } + case 'agent.activity_observed': + return e.p.openIncidents > before.openIncidents ? 'incident' : null; + case 'agent.dispatch_changed': + if (e.p.phase !== 'finished') return null; + switch (e.p.outcome) { + case 'completed': + return 'taskDone'; + case 'failed': + case 'timed_out': + case 'budget_exceeded': + case 'lost': + return 'incident'; + default: + return null; // cancelled: the viewer did it + } + case 'agent.session_changed': + if (e.p.phase !== 'ended') return null; + return e.p.outcome === 'succeeded' ? 'taskDone' : e.p.outcome === 'failed' ? 'incident' : null; + default: + return null; + } +} + +/** Fold one event into the tracker's agent state (the same fields `applyEvent` sets). */ +export function nextAgentState(prev: AgentAudioState | undefined, e: WorldEvent): AgentAudioState | undefined { + switch (e.type) { + case 'agent.housed': + return { ...HOUSED_AGENT }; + case 'agent.connection_changed': + return { ...(prev ?? HOUSED_AGENT), health: e.p.health, attention: e.p.attention }; + case 'agent.session_changed': + return e.p.phase === 'started' ? { ...(prev ?? HOUSED_AGENT), openIncidents: 0 } : prev; + case 'agent.activity_observed': + return { ...(prev ?? HOUSED_AGENT), openIncidents: e.p.openIncidents }; + default: + return prev; + } +} + +// ── Pacing ─────────────────────────────────────────────────────────── + +export interface CueLimits { + /** Minimum time between two cues; later ones wait in a short queue. */ + spacingMs: number; + queueMax: number; + /** A queued cue older than this is dropped (the moment has passed). */ + queueMaxAgeMs: number; + /** At most this many cues in any 60 s. */ + perMinute: number; + /** The same cue for the same agent waits this long. */ + cooldownMs: Readonly>; +} + +export const CUE_LIMITS: Readonly = { + spacingMs: 1200, + queueMax: 2, + queueMaxAgeMs: 4000, + perMinute: 6, + cooldownMs: { agentLive: 60_000, needsApproval: 30_000, taskDone: 20_000, incident: 15_000 }, +}; + +export type CueDecision = 'play' | 'queued' | 'cooldown' | 'rate' | 'queue_full' | 'suppressed' | 'hidden'; + +export interface PacedCue { + cue: EventCue; + agentId: string | null; + pan: number; + /** When it was offered (ms, the limiter's clock). */ + at: number; +} + +interface Queued extends PacedCue { + key: string; + /** The key's cooldown start before this cue took it (restored if it is dropped). */ + prevKeyAt: number | undefined; +} + +export class CueLimiter { + private lastPlay = -Infinity; + /** Times of cues played or queued within the last minute. */ + private recent: number[] = []; + private lastByKey = new Map(); + private queue: Queued[] = []; + private suppressUntil = -Infinity; + private hidden = false; + + constructor( + private readonly now: () => number, + private readonly limits: Readonly = CUE_LIMITS, + ) {} + + /** Ask to play a cue now. `play`: play it now; `queued`: `due()` will hand it out; anything else: dropped. */ + offer(cue: EventCue, agentId: string | null = null, pan = 0): CueDecision { + const t = this.now(); + if (this.hidden) return 'hidden'; + if (t < this.suppressUntil) return 'suppressed'; + const key = `${agentId ?? '*'}|${cue}`; + const last = this.lastByKey.get(key); + if (last !== undefined && t - last < this.limits.cooldownMs[cue]) return 'cooldown'; + this.recent = this.recent.filter((x) => t - x < 60_000); + if (this.recent.length >= this.limits.perMinute) return 'rate'; + if (this.queue.length === 0 && t - this.lastPlay >= this.limits.spacingMs) { + this.lastByKey.set(key, t); + this.recent.push(t); + this.lastPlay = t; + return 'play'; + } + if (this.queue.length >= this.limits.queueMax) { + // Make room by dropping the weakest waiting cue, unless the new one is weaker still. + let weakest = 0; + for (let i = 1; i < this.queue.length; i++) if (CUE_PRIORITY[this.queue[i]!.cue] <= CUE_PRIORITY[this.queue[weakest]!.cue]) weakest = i; + const w = this.queue[weakest]!; + if (CUE_PRIORITY[cue] <= CUE_PRIORITY[w.cue]) return 'queue_full'; + this.drop(weakest); + } + this.queue.push({ cue, agentId, pan, at: t, key, prevKeyAt: last }); + this.queue.sort((a, b) => CUE_PRIORITY[b.cue] - CUE_PRIORITY[a.cue] || a.at - b.at); + this.lastByKey.set(key, t); + this.recent.push(t); + return 'queued'; + } + + /** Waiting cues whose turn has come (at most one per spacing). Call it from a timer. */ + due(): PacedCue[] { + const t = this.now(); + if (this.hidden) { + this.clear(); + return []; + } + for (let i = this.queue.length - 1; i >= 0; i--) if (t - this.queue[i]!.at > this.limits.queueMaxAgeMs) this.drop(i); + if (this.queue.length === 0 || t - this.lastPlay < this.limits.spacingMs) return []; + const q = this.queue.shift()!; + this.lastPlay = t; + return [{ cue: q.cue, agentId: q.agentId, pan: q.pan, at: q.at }]; + } + + get waiting(): number { + return this.queue.length; + } + + /** No cues for `ms` (after a resync or reconnect: those events are not news). Clears the queue. */ + suppressFor(ms: number): void { + this.suppressUntil = Math.max(this.suppressUntil, this.now() + ms); + this.clear(); + } + + /** A hidden tab plays nothing and keeps nothing for later (no burst on return). */ + setHidden(hidden: boolean): void { + this.hidden = hidden; + if (hidden) this.clear(); + } + + private clear(): void { + for (let i = this.queue.length - 1; i >= 0; i--) this.drop(i); + } + + private drop(i: number): void { + const [q] = this.queue.splice(i, 1); + if (!q) return; + if (q.prevKeyAt === undefined) this.lastByKey.delete(q.key); + else this.lastByKey.set(q.key, q.prevKeyAt); + const r = this.recent.lastIndexOf(q.at); + if (r >= 0) this.recent.splice(r, 1); + } +} diff --git a/apps/studio/src/audio/engine.test.ts b/apps/studio/src/audio/engine.test.ts new file mode 100644 index 0000000..e9dbc41 --- /dev/null +++ b/apps/studio/src/audio/engine.test.ts @@ -0,0 +1,324 @@ +import { describe, expect, it } from 'vitest'; +import { AudioEngine, IDLE_SUSPEND_MS, LISTENER_STALE_MS, MAX_VOICES, PERSISTENT_NODES, installPageHooks } from './engine'; +import { MASTER_HEADROOM } from './graph'; +import { AUDIO_SETTINGS_KEY, DEFAULT_AUDIO_SETTINGS } from './settings'; +import { fakeEnv, type FakeAudioContext, type FakeEnv } from './testing'; +import type { AudioListener } from './types'; + +const listener = (p: Partial = {}): AudioListener => ({ + x: 48, + z: 36, + zoom01: 0.8, + az: 45, + entered: null, + level: 'street', + minute: 13 * 60, + night: 0, + paused: false, + sources: [ + { kind: 'murmur', id: 'spot_cafe_patio', gain: 0.8, pan: -0.3 }, + { kind: 'green', id: 'park_commons', gain: 0.7, pan: 0.4 }, + { kind: 'water', id: 'lm_fountain', gain: 0.6, pan: 0 }, + { kind: 'road', id: 'brook_road', gain: 0.5, pan: 0.6 }, + ], + ...p, +}); + +function setup(opts: Parameters[0] = {}) { + const env = fakeEnv(opts); + const audio = new AudioEngine(env); + const ctx = () => env.contexts[0] as FakeAudioContext | undefined; + /** Feed the listener for `ms` (as the town does, ~4 Hz) while time passes. */ + const live = (ms: number, l: AudioListener = listener()) => { + for (let t = 0; t < ms; t += 250) { + audio.setListener(l); + env.advance(250); + } + }; + return { env, audio, ctx, live }; +} + +describe('AudioEngine: autoplay and lifecycle (S1)', () => { + it('creates no AudioContext before the first gesture; nothing plays, nothing throws', () => { + const { env, audio } = setup(); + expect(audio.state).toBe('locked'); + audio.setListener(listener()); + audio.ui('select'); + audio.whoosh(500, 0.5); + expect(audio.cue('needsApproval', { agentId: 'agt_1' })).toBe(false); + env.advance(2000); + expect(env.contexts).toHaveLength(0); + expect(audio.debug()).toMatchObject({ created: false, voices: 0, persistentNodes: 0 }); + expect(audio.debug().lastCues.at(-1)?.result).toBe('off'); + }); + + it('the first gesture creates one context (idempotent), ≤ 40 persistent nodes, and the bed fades in over 2 s', () => { + const { env, audio, ctx } = setup(); + audio.unlock(); + audio.unlock(); + expect(env.contexts).toHaveLength(1); + expect(audio.state).toBe('running'); + expect(ctx()!.nodes).toHaveLength(PERSISTENT_NODES); + expect(PERSISTENT_NODES).toBeLessThanOrEqual(40); + // The ambient bus ramps from 0 to its volume (0.5² = 0.25) over 2 s. + const fade = ctx()!.params.flatMap((p) => p.events).find((e) => e.type === 'linear' && Math.abs(e.v - 0.25) < 1e-9); + expect(fade?.t).toBeCloseTo(2); + expect(audio.debug().buses).toMatchObject({ ambient: 0.25, ui: 0.16, events: 0.49 }); + expect(audio.debug().buses!.master).toBeCloseTo(MASTER_HEADROOM * 0.36, 3); + }); + + it('a gesture with sound off creates nothing; turning it on later (a click) starts it', () => { + const { env, audio } = setup({ stored: { enabled: false } }); + audio.unlock(); + expect(env.contexts).toHaveLength(0); + audio.update({ enabled: true }); + expect(env.contexts).toHaveLength(1); + expect(audio.state).toBe('running'); + }); + + it('unsupported browsers (no Web Audio) stay silent no-ops', () => { + const { env, audio } = setup({ supported: false }); + audio.unlock(); + audio.ui('open'); + expect(audio.cue('incident')).toBe(false); + expect(audio.state).toBe('unsupported'); + expect(env.contexts).toHaveLength(0); + }); + + it('a hidden tab fades the master to 0 in 150 ms and suspends at 300 ms; showing it resumes and fades in', () => { + const { env, audio, ctx } = setup(); + audio.unlock(); + audio.setHidden(true); + const master = () => audio.debug().buses!.master; + expect(master()).toBe(0); + const ramp = ctx()!.params.flatMap((p) => p.events).filter((e) => e.type === 'linear' && e.v === 0).at(-1); + expect(ramp?.t).toBeCloseTo(ctx()!.currentTime + 0.15); + env.advance(299); + expect(ctx()!.state).toBe('running'); + env.advance(2); + expect(ctx()!.state).toBe('suspended'); + expect(audio.state).toBe('suspended'); + // Hidden: cues are dropped, not saved for later. + expect(audio.cue('needsApproval', { agentId: 'agt_1' })).toBe(false); + expect(audio.debug().lastCues.at(-1)?.result).toBe('hidden'); + audio.setHidden(false); + expect(ctx()!.state).toBe('running'); + expect(master()).toBeCloseTo(MASTER_HEADROOM * 0.36, 3); + expect(audio.debug().cuesWaiting).toBe(0); + }); + + it('mute fades out and suspends; unmute resumes', () => { + const { env, audio, ctx } = setup(); + audio.unlock(); + const masterEvents = () => ctx()!.params.find((p) => p.events.some((e) => e.type === 'linear' && Math.abs(e.v - MASTER_HEADROOM * 0.36) < 1e-9))!.events; + const before = masterEvents().length; + audio.update({ enabled: false }); + expect(audio.debug().buses!.master).toBe(0); + // One 150 ms fade to 0, not overridden by the other buses' update. + const fades = masterEvents().slice(before).filter((e) => e.type === 'linear'); + expect(fades).toEqual([{ type: 'linear', v: 0, t: expect.closeTo(ctx()!.currentTime + 0.15, 5) }]); + env.advance(400); + expect(ctx()!.state).toBe('suspended'); + audio.ui('select'); + expect(audio.debug().voices).toBe(0); + audio.update({ enabled: true }); + expect(ctx()!.state).toBe('running'); + expect(audio.debug().buses!.master).toBeCloseTo(MASTER_HEADROOM * 0.36, 3); + }); +}); + +describe('AudioEngine: settings (S4)', () => { + it('persists each change in runloop.audio.v1, applies it to the buses and notifies subscribers', () => { + const { env, audio } = setup(); + audio.unlock(); + let calls = 0; + const off = audio.subscribe(() => calls++); + audio.update({ master: 0.5, ui: 1 }); + expect(JSON.parse(env.store.get(AUDIO_SETTINGS_KEY)!)).toEqual({ ...DEFAULT_AUDIO_SETTINGS, master: 0.5, ui: 1 }); + expect(audio.debug().buses).toMatchObject({ ui: 1 }); + expect(audio.debug().buses!.master).toBeCloseTo(MASTER_HEADROOM * 0.25, 3); + audio.update({ master: 0.5 }); // no change, no notification + audio.update({ reduce: true }); + expect(audio.debug().buses).toMatchObject({ ambient: 0, ui: 0.5 }); + expect(calls).toBe(2); + off(); + audio.update({ events: 0 }); + expect(calls).toBe(2); + expect(audio.getSettings().events).toBe(0); + // A new page reads them back. + const again = new AudioEngine({ ...env, contexts: [] } as FakeEnv); + expect(again.getSettings()).toEqual(audio.getSettings()); + }); + + it('reduce sound: no ambient layers, no hover, no agentLive cue; the other cues still play', () => { + const { audio, live } = setup({ stored: { reduce: true } }); + audio.unlock(); + live(3000); + expect(audio.debug().active).toEqual([]); + audio.ui('hover'); + expect(audio.debug().voices).toBe(0); + expect(audio.cue('agentLive', { agentId: 'agt_1' })).toBe(false); + expect(audio.debug().lastCues.at(-1)?.result).toBe('reduced'); + expect(audio.cue('needsApproval', { agentId: 'agt_1' })).toBe(true); + }); +}); + +describe('AudioEngine: voices and UI sounds', () => { + it('caps one-shot voices at 12, stealing the oldest UI voice; a cue still gets through', () => { + const { audio } = setup(); + audio.unlock(); + for (let i = 0; i < 20; i++) audio.ui('select'); + expect(audio.debug().voices).toBe(MAX_VOICES); + expect(audio.cue('incident', { agentId: 'agt_1' })).toBe(true); + expect(audio.debug().voices).toBe(MAX_VOICES); + }); + + it('hover ticks at most every 60 ms', () => { + const { env, audio } = setup(); + audio.unlock(); + audio.ui('hover'); + audio.ui('hover'); + env.clock.ms += 30; + audio.ui('hover'); + env.clock.ms += 40; + audio.ui('hover'); + expect(audio.debug().lastUi.filter((u) => u.sound === 'hover')).toHaveLength(2); + }); + + it('no whoosh under reduced motion', () => { + const { audio } = setup(); + audio.unlock(); + audio.setReducedMotion(true); + audio.whoosh(600, 0.4); + expect(audio.debug().voices).toBe(0); + audio.setReducedMotion(false); + audio.whoosh(600, 0.4); + expect(audio.debug().lastUi.at(-1)?.sound).toBe('whoosh'); + }); + + it('cues are paced by the limiter (cooldown per agent and cue)', () => { + const { env, audio } = setup(); + audio.unlock(); + expect(audio.cue('taskDone', { agentId: 'agt_1' })).toBe(true); + env.advance(2000); + expect(audio.cue('taskDone', { agentId: 'agt_1' })).toBe(false); + expect(audio.debug().lastCues.at(-1)?.result).toBe('cooldown'); + expect(audio.cue('taskDone', { agentId: 'agt_2' })).toBe(true); + // Within the 1.2 s spacing: queued, then played by the timer. + expect(audio.cue('incident', { agentId: 'agt_2' })).toBe(true); + expect(audio.debug().cuesWaiting).toBe(1); + env.advance(1500); + expect(audio.debug().cuesWaiting).toBe(0); + }); +}); + +describe('AudioEngine: the ambient bed follows the camera', () => { + it('a daytime street view near the café: breeze, birds, murmur, water, traffic', () => { + const { audio, live } = setup(); + audio.unlock(); + live(4000); + const d = audio.debug(); + expect(d.active).toEqual(expect.arrayContaining(['breeze', 'birds', 'murmur', 'water', 'traffic'])); + expect(d.active).not.toContain('crickets'); + expect(d.layers!.murmur).toBeGreaterThan(0.5); + expect(d.listener).toMatchObject({ fresh: true, level: 'street' }); + }); + + it('night brings crickets instead of birds; a paused world fades its life in ~1.5 s and keeps half the breeze', () => { + const { audio, live } = setup(); + audio.unlock(); + live(4000, listener({ night: 1 })); + expect(audio.debug().active).toContain('crickets'); + expect(audio.debug().layers!.birds).toBe(0); + const breeze = audio.debug().layers!.breeze; + live(2000, listener({ night: 1, paused: true })); + const p = audio.debug().layers!; + expect(p.crickets).toBeLessThan(0.05); + expect(p.murmur).toBeLessThan(0.05); + expect(p.traffic).toBeLessThan(0.02); + expect(p.breeze).toBeCloseTo(breeze / 2, 1); + }); + + it('inside a building: interior bed and room tone', () => { + const { audio, live } = setup(); + audio.unlock(); + live(2000, listener({ entered: 'bld_corner_cafe', level: 'interior' })); + expect(audio.debug().layers).toMatchObject({ interior: true }); + expect(audio.debug().active).toContain('room'); + }); + + it('agent buildings and lots never murmur', () => { + const { audio, live } = setup(); + audio.unlock(); + audio.setExcluded(['bld_agent_hq_1', 'lot_agent_1']); + live(3000, listener({ sources: [{ kind: 'murmur', id: 'bld_agent_hq_1', gain: 1, pan: 0 }] })); + expect(audio.debug().layers!.murmur).toBe(0); + }); + + it('without the town on screen the bed goes quiet, then the context idles until the next sound', () => { + const { env, audio, ctx, live } = setup(); + audio.unlock(); + live(3000); + env.advance(LISTENER_STALE_MS + 3000); + expect(audio.debug().active).toEqual([]); + env.advance(IDLE_SUSPEND_MS + 500); + expect(audio.debug().idle).toBe(true); + expect(ctx()!.state).toBe('suspended'); + audio.ui('open'); + expect(ctx()!.state).toBe('running'); + expect(audio.debug().voices).toBe(1); + }); + + it('birds are one-shot calls within the voice cap; nothing piles up over a long day', () => { + const { audio, live, ctx } = setup(); + audio.unlock(); + live(60_000, listener({ zoom01: 1, sources: [{ kind: 'green', id: 'park', gain: 1, pan: 0 }] })); + expect(audio.debug().voices).toBeLessThanOrEqual(MAX_VOICES); + const t = ctx()!.currentTime; + // Persistent sources (3 noise loops, the breeze LFO, the cricket oscillator) + a few voices at most. + expect(ctx()!.sourcesPlayingAt(t).length).toBeLessThanOrEqual(5 + 3 * MAX_VOICES); + }); +}); + +describe('page hooks', () => { + function page(hidden = false) { + const handlers = new Map void>>(); + const target = { + addEventListener: (type: string, fn: (e: Event) => void) => { + if (!handlers.has(type)) handlers.set(type, new Set()); + handlers.get(type)!.add(fn); + }, + removeEventListener: (type: string, fn: (e: Event) => void) => handlers.get(type)?.delete(fn), + }; + const fire = (type: string, ev: Partial = {}) => { + for (const fn of [...(handlers.get(type) ?? [])]) fn({ type, target: null, ...ev } as Event); + }; + const w = { ...target } as unknown as Window; + const docHandlers = new Map void>(); + const d = { visibilityState: hidden ? 'hidden' : 'visible', addEventListener: (t: string, fn: () => void) => docHandlers.set(t, fn) } as unknown as Document & { visibilityState: string }; + return { w, d, fire, handlers, docHandlers }; + } + + it('the first gesture unlocks (but not a click on the speaker toggle), later ones resume; visibility suspends; __runloopAudio reports', () => { + const env = fakeEnv(); + const audio = new AudioEngine(env); + const p = page(); + installPageHooks(audio, p.w, p.d); + const toggle = { closest: (s: string) => (s === '[data-audio-toggle]' ? toggle : null) }; + p.fire('pointerdown', { target: toggle as unknown as EventTarget }); + expect(env.contexts).toHaveLength(0); + p.fire('keydown'); + expect(env.contexts).toHaveLength(1); + // The browser suspends it (an interruption): the next gesture resumes it. + void (env.contexts[0] as FakeAudioContext).suspend(); + expect(audio.state).toBe('suspended'); + p.fire('pointerup'); + expect(audio.state).toBe('running'); + expect(env.contexts).toHaveLength(1); + (p.d as { visibilityState: string }).visibilityState = 'hidden'; + p.docHandlers.get('visibilitychange')!(); + expect((p.w as unknown as { __runloopAudio: { hidden: boolean } }).__runloopAudio.hidden).toBe(true); + const dbg = (p.w as unknown as { __runloopAudio: ReturnType }).__runloopAudio; + expect(dbg).toMatchObject({ created: true, persistentNodes: PERSISTENT_NODES }); + }); +}); diff --git a/apps/studio/src/audio/engine.ts b/apps/studio/src/audio/engine.ts new file mode 100644 index 0000000..5655d88 --- /dev/null +++ b/apps/studio/src/audio/engine.ts @@ -0,0 +1,545 @@ +/** + * The audio engine (contract §4b): one per page, created lazily. + * + * - No AudioContext before the first user gesture (browser autoplay policy): + * `unlock()` creates it, and only when sound is on. The bed fades in over 2 s. + * - A 250 ms timer follows the camera's listener and schedules ahead on the + * audio clock; without a fresh listener (no town on screen) the bed is + * silent and the context suspends itself until the next sound. + * - Hidden tab: master → 0 in 150 ms, `suspend()` at 300 ms, cues dropped (no + * burst on return); visible again: `resume()`, fade in over 600 ms. + * - ≤ 40 persistent nodes; ≤ 12 one-shot voices (the oldest UI voice is stolen). + * - `window.__runloopAudio` (dev and prod, read-only) reports state, layers, + * bus levels and the last cues for automated checks. + */ +import { Soundscape, SOUNDSCAPE_NODES, type BedLevels } from './ambient'; +import { CueLimiter, type CueDecision } from './cues'; +import { GRAPH_NODES, applyBuses, buildGraph, busTargets, rampTo, type AudioGraph } from './graph'; +import { SILENT_MIX, ambientMix } from './mix'; +import { AUDIO_SETTINGS_KEY, DEFAULT_AUDIO_SETTINGS, loadAudioSettings, mergeAudioSettings, saveAudioSettings, sameAudioSettings } from './settings'; +import { birdVoice, cueVoice, makeBuffers, seededRng, uiVoice, whooshVoice, type Rng, type Synth, type Voice, type VoiceKind } from './synth'; +import type { AudioListener, AudioSettings, AudioState, EventCue, RunloopAudio, UiSound } from './types'; + +export const MAX_VOICES = 12; +export const PERSISTENT_NODES = GRAPH_NODES + SOUNDSCAPE_NODES; +export const TICK_MS = 250; +/** Without a listener update for this long, the bed goes quiet (no town on screen). */ +export const LISTENER_STALE_MS = 3000; +/** Quiet this long (no bed, no voices) → suspend until the next sound. */ +export const IDLE_SUSPEND_MS = 5000; +const HOVER_MIN_MS = 60; +const BED_FADE_IN_S = 2; + +type Timer = ReturnType; + +/** What the engine needs from the page (a fake in tests). */ +export interface EngineEnv { + /** Web Audio exists here. */ + supported: boolean; + createContext(): AudioContext | null; + storage: Pick | null; + /** Wall clock, ms. */ + now(): number; + setInterval(fn: () => void, ms: number): Timer; + clearInterval(id: Timer): void; + setTimeout(fn: () => void, ms: number): Timer; + clearTimeout(id: Timer): void; + rng?: Rng; +} + +export interface CueRecord { + cue: EventCue; + agentId: string | null; + /** ms, the engine's clock. */ + at: number; + /** played / queued, or why not. */ + result: CueDecision | 'off' | 'reduced'; +} + +export interface UiRecord { + sound: UiSound | 'whoosh'; + at: number; +} + +export interface AudioDebug { + state: AudioState; + created: boolean; + gestureSeen: boolean; + hidden: boolean; + idle: boolean; + reducedMotion: boolean; + settings: AudioSettings; + listener: { fresh: boolean; ageMs: number; level: AudioListener['level']; zoom01: number; night: number; paused: boolean; entered: string | null; sources: number } | null; + layers: BedLevels | null; + /** Layers audible now. */ + active: string[]; + /** Current bus gains (linear). */ + buses: { master: number; ambient: number; ui: number; events: number } | null; + voices: number; + persistentNodes: number; + cuesWaiting: number; + lastCues: CueRecord[]; + lastUi: UiRecord[]; +} + +export class AudioEngine implements RunloopAudio { + readonly limiter: CueLimiter; + private supported: boolean; + private ctx: AudioContext | null = null; + private graph: AudioGraph | null = null; + private scape: Soundscape | null = null; + private synth: Synth | null = null; + private readonly rng: Rng; + private settings: AudioSettings; + private readonly subscribers = new Set<() => void>(); + private gestureSeen = false; + private hidden = false; + private idle = false; + private reducedMotion = false; + private listener: AudioListener | null = null; + private listenerAt = -Infinity; + private exclude = new Set(); + private voices: Voice[] = []; + private timer: Timer | null = null; + private pendingSuspend: Timer | null = null; + private lastTick = 0; + private quietSince: number | null = null; + private lastHover = -Infinity; + private cues: CueRecord[] = []; + private uis: UiRecord[] = []; + + constructor(private readonly env: EngineEnv) { + this.supported = env.supported; + this.rng = env.rng ?? seededRng(Math.floor(env.now()) ^ 0x5bd1e995); + this.settings = loadAudioSettings(env.storage); + this.limiter = new CueLimiter(() => env.now()); + } + + get state(): AudioState { + if (!this.supported) return 'unsupported'; + if (!this.ctx) return 'locked'; + return this.ctx.state === 'running' ? 'running' : 'suspended'; + } + + // ── Settings ── + + getSettings = (): AudioSettings => this.settings; + + subscribe = (fn: () => void): (() => void) => { + this.subscribers.add(fn); + return () => this.subscribers.delete(fn); + }; + + update = (p: Partial): void => { + const next = mergeAudioSettings(this.settings, p); + if (sameAudioSettings(next, this.settings)) return; + saveAudioSettings(next, this.env.storage); + this.applySettings(next); + }; + + /** Another tab changed the stored settings. */ + reloadSettings(): void { + const next = loadAudioSettings(this.env.storage); + if (!sameAudioSettings(next, this.settings)) this.applySettings(next); + } + + private applySettings(next: AudioSettings): void { + const prev = this.settings; + this.settings = next; + if (next.enabled && !prev.enabled && this.gestureSeen) this.ensureContext(); + const { ctx, graph } = this; + if (ctx && graph) { + const now = ctx.currentTime; + this.cancelSuspend(); + if (!next.enabled) { + // Fade out, then suspend; the other buses follow the settings quietly. + rampTo(graph.master.gain, 0, now, 0.15); + this.suspendSoon(() => !this.settings.enabled); + applyBuses(graph, this.targets(), now, 0.08, { master: false }); + } else { + if (!prev.enabled) this.run(); + applyBuses(graph, this.targets(), now, prev.enabled ? 0.08 : 0.4); + } + } + this.emit(); + } + + // ── Lifecycle ── + + /** + * A user gesture: create the context when sound is on, or resume one the + * browser suspended (autoplay, an interruption). Idempotent and cheap; it + * leaves an idle context asleep (the next sound wakes it). + */ + unlock = (): void => { + this.gestureSeen = true; + if (!this.supported || !this.settings.enabled) return; + const fresh = !this.ctx; + if (!this.ensureContext()) return; + if (!fresh && this.ctx!.state !== 'running' && !this.hidden && !this.idle) this.run(); + }; + + private ensureContext(): boolean { + if (this.ctx) return true; + if (!this.supported) return false; + let ctx: AudioContext | null = null; + try { + ctx = this.env.createContext(); + } catch { + ctx = null; + } + if (!ctx) { + this.supported = false; + this.emit(); + return false; + } + this.ctx = ctx; + const buffers = makeBuffers(ctx, this.rng); + const graph = buildGraph(ctx, buffers); + this.graph = graph; + this.synth = { ctx, buffers, rng: this.rng }; + this.scape = new Soundscape(graph, this.rng, (t, species, pan, peak) => { + this.spawn('bird', (s) => birdVoice(s, graph.birds, t, species, pan, peak)); + }); + const now = ctx.currentTime; + this.scape.start(now); + const tg = this.targets(); + rampTo(graph.ui.gain, tg.ui, now, 0); + rampTo(graph.events.gain, tg.events, now, 0); + rampTo(graph.master.gain, tg.master, now, 0.3); + rampTo(graph.ambient.gain, tg.ambient, now, BED_FADE_IN_S); + rampTo(graph.birdsSend.gain, tg.birdsSend, now, BED_FADE_IN_S); + ctx.onstatechange = () => this.emit(); + this.lastTick = now; + if (ctx.state !== 'running' && !this.hidden) void ctx.resume().catch(() => undefined); + this.startTimer(); + this.emit(); + return true; + } + + /** Resume and tick (sound on, tab visible). */ + private run(): void { + const ctx = this.ctx; + if (!ctx || !this.settings.enabled || this.hidden) return; + this.idle = false; + this.quietSince = null; + if (ctx.state !== 'running') void ctx.resume().catch(() => undefined); + this.startTimer(); + } + + private suspendSoon(still: () => boolean): void { + this.cancelSuspend(); + this.pendingSuspend = this.env.setTimeout(() => { + this.pendingSuspend = null; + if (!still() || !this.ctx) return; + this.stopTimer(); + void this.ctx.suspend().catch(() => undefined); + }, 300); + } + + private cancelSuspend(): void { + if (this.pendingSuspend) this.env.clearTimeout(this.pendingSuspend); + this.pendingSuspend = null; + } + + private startTimer(): void { + this.timer ??= this.env.setInterval(() => this.tick(), TICK_MS); + } + + private stopTimer(): void { + if (this.timer) this.env.clearInterval(this.timer); + this.timer = null; + } + + /** The page was hidden or shown. */ + setHidden(hidden: boolean): void { + if (hidden === this.hidden) return; + this.hidden = hidden; + this.limiter.setHidden(hidden); + const { ctx, graph } = this; + if (ctx && graph) { + if (hidden) { + rampTo(graph.master.gain, 0, ctx.currentTime, 0.15); + this.suspendSoon(() => this.hidden); + } else if (this.settings.enabled) { + this.cancelSuspend(); + this.run(); + rampTo(graph.master.gain, this.targets().master, ctx.currentTime, 0.6); + } + } + this.emit(); + } + + /** One scheduling step (the timer calls it; renders and tests may too). */ + tick(): void { + const { ctx, graph, scape } = this; + if (!ctx || !graph || !scape) return; + const now = ctx.currentTime; + const dt = now - this.lastTick; + this.lastTick = now; + const fresh = this.listenerFresh(); + const mix = fresh ? ambientMix(this.listener, { reduce: this.settings.reduce, exclude: this.exclude }) : SILENT_MIX; + const wasInside = scape.inside; + scape.update(now, mix, dt); + if (scape.inside !== wasInside) rampTo(graph.birdsSend.gain, this.targets().birdsSend, now, 0.3); + for (const c of this.limiter.due()) this.playCue(c.cue, c.pan); + this.voices = this.voices.filter((v) => v.end > now); + // Nothing to hear (no town on screen, no voices): suspend until the next sound. + const quiet = !fresh && this.voices.length === 0 && this.limiter.waiting === 0; + const wall = this.env.now(); + if (!quiet) this.quietSince = null; + else if (this.quietSince === null) this.quietSince = wall; + else if (wall - this.quietSince > IDLE_SUSPEND_MS && ctx.state === 'running') { + this.idle = true; + this.stopTimer(); + void ctx.suspend().catch(() => undefined); + } + } + + private wake(): void { + if (this.idle) this.run(); + } + + private listenerFresh(): boolean { + return this.listener !== null && this.env.now() - this.listenerAt < LISTENER_STALE_MS; + } + + private targets() { + return busTargets(this.settings, { audible: !this.hidden, interior: this.scape?.inside ?? false }); + } + + private canPlay(): boolean { + return !!this.ctx && !!this.graph && this.settings.enabled && !this.hidden; + } + + // ── Voices ── + + /** Start a one-shot voice; at the cap, steal the oldest UI voice (then a bird call; events only for events). */ + private spawn(kind: VoiceKind, build: (s: Synth) => Voice): boolean { + const { ctx, synth } = this; + if (!ctx || !synth) return false; + const now = ctx.currentTime; + this.voices = this.voices.filter((v) => v.end > now); + if (this.voices.length >= MAX_VOICES) { + const victim = this.voices.find((v) => v.kind === 'ui') ?? this.voices.find((v) => v.kind === 'bird') ?? (kind === 'event' ? this.voices.find((v) => v.kind === 'event') : undefined); + if (!victim) return false; + victim.out.gain.cancelScheduledValues(now); + victim.out.gain.setTargetAtTime(0, now, 0.01); + for (const s of victim.sources) { + try { + s.stop(now + 0.06); + } catch { + // already stopped + } + } + this.voices.splice(this.voices.indexOf(victim), 1); + } + this.voices.push(build(synth)); + return true; + } + + ui = (sound: UiSound): void => { + if (!this.canPlay() || this.settings.ui <= 0) return; + if (sound === 'hover') { + if (this.settings.reduce) return; + const t = this.env.now(); + if (t - this.lastHover < HOVER_MIN_MS) return; + this.lastHover = t; + } + this.wake(); + const graph = this.graph!; + const t0 = this.ctx!.currentTime + 0.005; + if (this.spawn('ui', (s) => uiVoice(s, graph.ui, t0, sound))) this.recordUi(sound); + }; + + whoosh = (durationMs: number, pan = 0): void => { + if (this.reducedMotion || !this.canPlay() || this.settings.ui <= 0 || !(durationMs > 0)) return; + this.wake(); + const graph = this.graph!; + const t0 = this.ctx!.currentTime + 0.01; + const p = Number.isFinite(pan) ? Math.max(-1, Math.min(1, pan)) : 0; + if (this.spawn('ui', (s) => whooshVoice(s, graph.ui, t0, durationMs / 1000, p))) this.recordUi('whoosh'); + }; + + cue = (cue: EventCue, opts: { agentId?: string; pan?: number } = {}): boolean => { + const agentId = opts.agentId ?? null; + const pan = Number.isFinite(opts.pan) ? Math.max(-1, Math.min(1, opts.pan!)) : 0; + if (!this.ctx || !this.settings.enabled || this.settings.events <= 0) return this.recordCue(cue, agentId, 'off'); + if (this.settings.reduce && cue === 'agentLive') return this.recordCue(cue, agentId, 'reduced'); + const d = this.limiter.offer(cue, agentId, pan); + this.recordCue(cue, agentId, d); + if (d === 'play') this.playCue(cue, pan); + return d === 'play' || d === 'queued'; + }; + + private playCue(cue: EventCue, pan: number): void { + if (!this.canPlay()) return; + this.wake(); + const graph = this.graph!; + const t0 = this.ctx!.currentTime + 0.01; + this.spawn('event', (s) => cueVoice(s, graph.events, t0, cue, pan)); + } + + /** The settings' Preview buttons (a user action; not an event cue, so no pacing). */ + preview(what: 'ui' | 'events' | 'ambient'): void { + this.unlock(); + if (!this.canPlay()) return; + this.wake(); + const graph = this.graph!; + const t0 = this.ctx!.currentTime + 0.01; + if (what === 'ui') this.spawn('ui', (s) => uiVoice(s, graph.ui, t0, 'confirm')); + else if (what === 'events') this.spawn('event', (s) => cueVoice(s, graph.events, t0, 'taskDone')); + } + + /** One cue as a preview (the sound board; a user action, not an event). */ + previewCue(cue: EventCue): void { + this.unlock(); + if (!this.canPlay()) return; + this.wake(); + const graph = this.graph!; + const t0 = this.ctx!.currentTime + 0.01; + this.spawn('event', (s) => cueVoice(s, graph.events, t0, cue)); + } + + // ── World ── + + setListener = (l: AudioListener): void => { + this.listener = l; + this.listenerAt = this.env.now(); + this.wake(); + }; + + setReducedMotion = (v: boolean): void => { + this.reducedMotion = v; + }; + + /** Places that may never murmur (agent buildings and lots). */ + setExcluded(ids: Iterable): void { + this.exclude = new Set(ids); + } + + /** No cues for `ms` (after a resync or reconnect). */ + suppressCues(ms: number): void { + this.limiter.suppressFor(ms); + } + + // ── Debug ── + + private recordCue(cue: EventCue, agentId: string | null, result: CueRecord['result']): false { + this.cues.push({ cue, agentId, at: Math.round(this.env.now()), result }); + if (this.cues.length > 20) this.cues.shift(); + return false; + } + + private recordUi(sound: UiRecord['sound']): void { + this.uis.push({ sound, at: Math.round(this.env.now()) }); + if (this.uis.length > 20) this.uis.shift(); + } + + debug(): AudioDebug { + const l = this.listener; + const layers = this.scape?.levels ?? null; + const g = this.graph; + const r = (v: number) => Math.round(v * 10000) / 10000; + return { + state: this.state, + created: this.ctx !== null, + gestureSeen: this.gestureSeen, + hidden: this.hidden, + idle: this.idle, + reducedMotion: this.reducedMotion, + settings: { ...this.settings }, + listener: l + ? { fresh: this.listenerFresh(), ageMs: Math.round(this.env.now() - this.listenerAt), level: l.level, zoom01: l.zoom01, night: l.night, paused: l.paused, entered: l.entered, sources: l.sources.length } + : null, + layers, + active: layers ? (['breeze', 'wind', 'birds', 'crickets', 'murmur', 'water', 'traffic', 'room'] as const).filter((k) => layers[k] > 0.01) : [], + buses: g ? { master: r(g.master.gain.value), ambient: r(g.ambient.gain.value), ui: r(g.ui.gain.value), events: r(g.events.gain.value) } : null, + voices: this.voices.length, + persistentNodes: this.ctx ? PERSISTENT_NODES : 0, + cuesWaiting: this.limiter.waiting, + lastCues: this.cues.slice(-10), + lastUi: this.uis.slice(-10), + }; + } + + private emit(): void { + for (const fn of [...this.subscribers]) fn(); + } +} + +// ── The page's engine ──────────────────────────────────────────────── + +/** Server rendering and browsers without Web Audio: everything is a no-op. */ +const SILENT: RunloopAudio = Object.freeze({ + state: 'unsupported' as const, + unlock: () => undefined, + ui: () => undefined, + whoosh: () => undefined, + cue: () => false, + setListener: () => undefined, + setReducedMotion: () => undefined, + getSettings: () => DEFAULT_AUDIO_SETTINGS, + update: () => undefined, + subscribe: () => () => undefined, +}); + +let engine: AudioEngine | null = null; + +function browserEnv(): EngineEnv { + const w = window as Window & { webkitAudioContext?: typeof AudioContext }; + const Ctor = typeof AudioContext === 'function' ? AudioContext : w.webkitAudioContext; + let storage: Storage | null = null; + try { + storage = w.localStorage; + } catch { + storage = null; + } + return { + supported: typeof Ctor === 'function', + createContext: () => (Ctor ? new Ctor({ latencyHint: 'interactive' }) : null), + storage, + now: () => performance.now(), + setInterval: (fn, ms) => setInterval(fn, ms), + clearInterval: (id) => clearInterval(id), + setTimeout: (fn, ms) => setTimeout(fn, ms), + clearTimeout: (id) => clearTimeout(id), + }; +} + +/** + * Gestures unlock sound (and resume it after the browser suspends it); the + * tab's visibility suspends it; other tabs' settings apply here. + */ +export function installPageHooks(e: AudioEngine, w: Window, d: Document): void { + const gestures = ['pointerdown', 'pointerup', 'keydown', 'touchend'] as const; + const onGesture = (ev: Event) => { + if (e.state === 'running' || e.state === 'unsupported') return; + const target = ev.target as { closest?: (s: string) => unknown } | null; + // The speaker toggle handles its own click (so a first click on it mutes rather than starts sound). + if (target && typeof target.closest === 'function' && target.closest('[data-audio-toggle]')) return; + e.unlock(); + }; + for (const g of gestures) w.addEventListener(g, onGesture, { capture: true, passive: true }); + e.setHidden(d.visibilityState === 'hidden'); + d.addEventListener('visibilitychange', () => e.setHidden(d.visibilityState === 'hidden')); + w.addEventListener('storage', (ev) => { + if (ev.key === AUDIO_SETTINGS_KEY) e.reloadSettings(); + }); + Object.defineProperty(w, '__runloopAudio', { configurable: true, get: () => e.debug() }); +} + +/** The page's engine (null on the server). */ +export function getAudioEngine(): AudioEngine | null { + if (typeof window === 'undefined' || typeof document === 'undefined') return null; + if (!engine) { + engine = new AudioEngine(browserEnv()); + installPageHooks(engine, window, document); + } + return engine; +} + +/** The soundscape (a singleton; SSR-safe no-op on the server). */ +export function getAudio(): RunloopAudio { + return getAudioEngine() ?? SILENT; +} diff --git a/apps/studio/src/audio/graph.ts b/apps/studio/src/audio/graph.ts new file mode 100644 index 0000000..f2c05c4 --- /dev/null +++ b/apps/studio/src/audio/graph.ts @@ -0,0 +1,134 @@ +/** + * The mixing graph (contract §4b): + * + * ambient layers → outdoor (−9 dB inside) → outdoorLp (700 Hz inside) ┐ + * room tone ───────────────────────────────────────────────────────── ambient ┐ + * birds ── outdoor; birds → birdsSend (0.25) → reverb │ + * UI voices → ui ── uiSend (0.1) → reverb ├→ master → compressor (−18 dB, 3:1) → out + * cue voices → events ── eventsSend (0.2) → reverb │ + * reverb (generated 1.6 s impulse) ────────────────────────────────────────── ┘ + * + * Bus gains come from the settings (`busTargets`); the birds' reverb send + * follows the ambient volume and the interior so it never outlives them. + */ +import { sliderGain } from './settings'; +import type { Buffers } from './synth'; +import type { AudioSettings } from './types'; + +export interface AudioGraph { + ctx: BaseAudioContext; + buffers: Buffers; + master: GainNode; + compressor: DynamicsCompressorNode; + ambient: GainNode; + outdoor: GainNode; + outdoorLp: BiquadFilterNode; + birds: GainNode; + birdsSend: GainNode; + ui: GainNode; + uiSend: GainNode; + events: GainNode; + eventsSend: GainNode; + reverb: ConvolverNode; +} + +/** Persistent nodes the graph itself creates (the soundscape adds its own). */ +export const GRAPH_NODES = 12; + +export const BIRDS_SEND = 0.25; +export const UI_SEND = 0.1; +export const EVENTS_SEND = 0.2; +/** Inside a building the birds' reverb (which skips the interior lowpass) almost goes. */ +export const BIRDS_SEND_INSIDE = 0.15; +/** 1 dB of headroom on the master: the busiest moment at full volume stays under 0.9 (render check). */ +export const MASTER_HEADROOM = 0.89; +/** Outdoor lowpass: open outside, 700 Hz inside a building. */ +export const OUTDOOR_OPEN_HZ = 18000; +export const OUTDOOR_INSIDE_HZ = 700; + +export function buildGraph(ctx: BaseAudioContext, buffers: Buffers, out: AudioNode = ctx.destination): AudioGraph { + const gain = (v: number) => { + const g = ctx.createGain(); + g.gain.value = v; + return g; + }; + const master = gain(0); + const compressor = ctx.createDynamicsCompressor(); + compressor.threshold.value = -18; + compressor.ratio.value = 3; + compressor.knee.value = 4; + compressor.attack.value = 0.002; + compressor.release.value = 0.25; + master.connect(compressor).connect(out); + + const reverb = ctx.createConvolver(); + reverb.buffer = buffers.impulse; + reverb.connect(master); + + const ambient = gain(0); + ambient.connect(master); + const outdoor = gain(1); + const outdoorLp = ctx.createBiquadFilter(); + outdoorLp.type = 'lowpass'; + outdoorLp.frequency.value = OUTDOOR_OPEN_HZ; + outdoorLp.Q.value = 0.5; + outdoor.connect(outdoorLp).connect(ambient); + + const birds = gain(1); + birds.connect(outdoor); + const birdsSend = gain(0); + birds.connect(birdsSend).connect(reverb); + + const ui = gain(0); + ui.connect(master); + const uiSend = gain(UI_SEND); + ui.connect(uiSend).connect(reverb); + + const events = gain(0); + events.connect(master); + const eventsSend = gain(EVENTS_SEND); + events.connect(eventsSend).connect(reverb); + + return { ctx, buffers, master, compressor, ambient, outdoor, outdoorLp, birds, birdsSend, ui, uiSend, events, eventsSend, reverb }; +} + +export interface BusTargets { + master: number; + ambient: number; + birdsSend: number; + ui: number; + events: number; +} + +/** + * Bus gains for the settings. `audible` = sound on and the tab visible. + * Reduce sound: no ambient bed, UI −6 dB (cues are filtered by the engine). + */ +export function busTargets(s: AudioSettings, opts: { audible: boolean; interior?: boolean }): BusTargets { + const ambient = s.reduce ? 0 : sliderGain(s.ambient); + return { + master: opts.audible && s.enabled ? MASTER_HEADROOM * sliderGain(s.master) : 0, + ambient, + birdsSend: BIRDS_SEND * ambient * (opts.interior ? BIRDS_SEND_INSIDE : 1), + ui: sliderGain(s.ui) * (s.reduce ? 0.5 : 1), + events: sliderGain(s.events), + }; +} + +/** Glide a param to `v` over `seconds` from its current value (0 = now). */ +export function rampTo(p: AudioParam, v: number, at: number, seconds: number): void { + p.cancelScheduledValues(at); + p.setValueAtTime(p.value, at); + if (seconds <= 0) p.setValueAtTime(v, at); + else p.linearRampToValueAtTime(v, at + seconds); +} + +export function applyBuses(g: AudioGraph, t: BusTargets, at: number, seconds: number, opts: { master?: boolean; ambient?: boolean } = {}): void { + if (opts.master !== false) rampTo(g.master.gain, t.master, at, seconds); + if (opts.ambient !== false) { + rampTo(g.ambient.gain, t.ambient, at, seconds); + rampTo(g.birdsSend.gain, t.birdsSend, at, seconds); + } + rampTo(g.ui.gain, t.ui, at, seconds); + rampTo(g.events.gain, t.events, at, seconds); +} diff --git a/apps/studio/src/audio/index.ts b/apps/studio/src/audio/index.ts new file mode 100644 index 0000000..3e660fa --- /dev/null +++ b/apps/studio/src/audio/index.ts @@ -0,0 +1,18 @@ +/** + * Runloop's soundscape (contract §4b), synthesized with the Web Audio API: + * no audio files, no dependencies. See README.md. + * + * getAudio() the page's engine (SSR-safe no-op on the server) + * attachWorldAudio(store) real agent events → cues (once, on `/`) + * classifyCue(event, prev) pure: which event earns which cue (never simulated activity) + * useAudioSettings() [settings, update] + * the HUD speaker toggle + * the Esc menu's Sound tab + */ +export type { AgentAudioState, AudioListener, AudioSettings, AudioSourceKind, AudioState, EventCue, RunloopAudio, UiSound } from './types'; +export { EVENT_CUES, UI_SOUNDS } from './types'; +export { getAudio } from './engine'; +export { attachWorldAudio } from './world'; +export { classifyCue } from './cues'; +export { AUDIO_SETTINGS_KEY, DEFAULT_AUDIO_SETTINGS } from './settings'; +export { SoundSettings, SoundToggle, useAudioSettings, useAudioState } from './react'; diff --git a/apps/studio/src/audio/mix.test.ts b/apps/studio/src/audio/mix.test.ts new file mode 100644 index 0000000..10f8864 --- /dev/null +++ b/apps/studio/src/audio/mix.test.ts @@ -0,0 +1,145 @@ +import { describe, expect, it } from 'vitest'; +import { INTERIOR_OUTDOOR_GAIN, ROOM_TONE, SILENT_MIX, ambientMix, smoothToward } from './mix'; +import type { AudioListener } from './types'; + +const base = (p: Partial = {}): AudioListener => ({ + x: 48, + z: 36, + zoom01: 0.5, + az: 45, + entered: null, + level: 'district', + minute: 12 * 60, + night: 0, + paused: false, + sources: [], + ...p, +}); + +describe('ambientMix', () => { + it('is silent without a listener (no town on screen) and under "reduce sound"', () => { + expect(ambientMix(null)).toEqual(SILENT_MIX); + expect(ambientMix(base({ sources: [{ kind: 'murmur', id: 'spot_cafe_patio', gain: 1, pan: 0 }] }), { reduce: true })).toEqual(SILENT_MIX); + }); + + it('breeze and high wind follow zoom: wide views are airier', () => { + const town = ambientMix(base({ zoom01: 0 })); + const street = ambientMix(base({ zoom01: 1 })); + expect(town.breeze).toBeCloseTo(0.6); + expect(street.breeze).toBeCloseTo(0.35); + expect(town.wind).toBeCloseTo(0.25); + expect(street.wind).toBe(0); + }); + + it('birds by day, crickets by night, cross-fading continuously with night', () => { + const green = [{ kind: 'green' as const, id: 'park', gain: 0.8, pan: 0 }]; + let lastBirds = Infinity; + let lastCrickets = -Infinity; + for (let n = 0; n <= 1.0001; n += 0.1) { + const m = ambientMix(base({ night: n, sources: green })); + expect(m.birdsRate).toBeLessThanOrEqual(lastBirds); + expect(m.crickets).toBeGreaterThanOrEqual(lastCrickets); + lastBirds = m.birdsRate; + lastCrickets = m.crickets; + } + expect(ambientMix(base({ night: 0, sources: green })).crickets).toBe(0); + expect(ambientMix(base({ night: 1, sources: green })).birdsRate).toBe(0); + // calls/s = 0.5·day·green·(0.25+0.75z), green = 0.1 floor + 0.8 + expect(ambientMix(base({ night: 0, zoom01: 1, sources: green })).birdsRate).toBeCloseTo(0.5 * 0.9); + }); + + it('greenery and fields bring more birds than rooftops', () => { + const roofs = ambientMix(base()); + const park = ambientMix(base({ sources: [{ kind: 'green', id: 'park', gain: 0.6, pan: 0 }] })); + const fields = ambientMix(base({ sources: [{ kind: 'field', id: 'field_1', gain: 0.6, pan: 0 }] })); + expect(roofs.birdsRate).toBeGreaterThan(0); + expect(fields.birdsRate).toBeGreaterThan(roofs.birdsRate); + expect(park.birdsRate).toBeGreaterThan(fields.birdsRate); + }); + + it('murmur comes only from murmur sources, grows with zoom, is capped, pans to its side, and never from an excluded (agent) place', () => { + const src = [ + { kind: 'murmur' as const, id: 'spot_cafe_patio', gain: 0.6, pan: -0.5 }, + { kind: 'murmur' as const, id: 'spot_fountain_rim', gain: 0.9, pan: -0.2 }, + { kind: 'water' as const, id: 'pond', gain: 1, pan: 0.8 }, + { kind: 'road' as const, id: 'brook_road', gain: 1, pan: 0.5 }, + ]; + const near = ambientMix(base({ zoom01: 1, sources: src })); + expect(near.murmur).toBe(1); + expect(near.murmurPan).toBeLessThan(0); + const far = ambientMix(base({ zoom01: 0, sources: src })); + expect(far.murmur).toBeCloseTo(0.2); + expect(ambientMix(base({ sources: src.filter((s) => s.kind !== 'murmur') })).murmur).toBe(0); + const excluded = ambientMix(base({ zoom01: 1, sources: src }), { exclude: new Set(['spot_cafe_patio', 'spot_fountain_rim']) }); + expect(excluded.murmur).toBe(0); + }); + + it('water is the nearest water source, heard close up; traffic is a distant hum, quieter at night', () => { + const src = [ + { kind: 'water' as const, id: 'brook', gain: 0.4, pan: -1 }, + { kind: 'water' as const, id: 'fountain', gain: 0.9, pan: 0.3 }, + { kind: 'road' as const, id: 'westway', gain: 0.7, pan: -0.6 }, + ]; + const m = ambientMix(base({ zoom01: 1, sources: src })); + expect(m.water).toBeCloseTo(0.9); + expect(m.waterPan).toBeCloseTo(0.3); + expect(m.trafficPan).toBeCloseTo(-0.6); + expect(ambientMix(base({ zoom01: 0, sources: src })).water).toBe(0); + expect(ambientMix(base({ zoom01: 1, night: 0 })).traffic).toBeCloseTo(0.15); + expect(ambientMix(base({ zoom01: 1, night: 1 })).traffic).toBeCloseTo(0.15 * 0.4); + expect(ambientMix(base({ zoom01: 0, night: 0 })).traffic).toBeCloseTo(0.35); + }); + + it('a paused world goes quiet: no murmur, traffic, birds or crickets; the breeze stays at half', () => { + const src = [ + { kind: 'murmur' as const, id: 'spot_cafe_patio', gain: 1, pan: 0 }, + { kind: 'green' as const, id: 'park', gain: 1, pan: 0 }, + { kind: 'water' as const, id: 'pond', gain: 1, pan: 0 }, + ]; + for (const night of [0, 1]) { + const run = ambientMix(base({ night, sources: src })); + const paused = ambientMix(base({ night, sources: src, paused: true })); + expect(paused.murmur).toBe(0); + expect(paused.traffic).toBe(0); + expect(paused.birdsRate).toBe(0); + expect(paused.crickets).toBe(0); + expect(paused.breeze).toBeCloseTo(run.breeze / 2); + expect(paused.water).toBe(run.water); + } + }); + + it('inside a building: interior bed and room tone', () => { + const m = ambientMix(base({ entered: 'bld_corner_cafe', level: 'interior' })); + expect(m.interior).toBe(true); + expect(m.room).toBe(ROOM_TONE); + expect(ambientMix(base()).interior).toBe(false); + expect(20 * Math.log10(INTERIOR_OUTDOOR_GAIN)).toBeCloseTo(-9); + }); + + it('keeps every level in range for any listener, even malformed numbers', () => { + let seed = 7; + const rnd = () => ((seed = (seed * 16807) % 2147483647) / 2147483647) * 2 - 0.5; + for (let i = 0; i < 300; i++) { + const kinds = ['murmur', 'water', 'green', 'road', 'field'] as const; + const sources = Array.from({ length: 6 }, (_, j) => ({ kind: kinds[j % 5]!, id: `s${j}`, gain: rnd() * 2, pan: rnd() * 3 })); + const m = ambientMix(base({ zoom01: rnd() * 2, night: i === 0 ? Number.NaN : rnd() * 2, paused: i % 3 === 0, sources })); + for (const k of ['breeze', 'wind', 'crickets', 'murmur', 'water', 'traffic', 'room', 'day'] as const) { + expect(m[k], k).toBeGreaterThanOrEqual(0); + expect(m[k], k).toBeLessThanOrEqual(1); + } + expect(m.birdsRate).toBeLessThanOrEqual(0.5); + for (const k of ['murmurPan', 'waterPan', 'trafficPan'] as const) expect(Math.abs(m[k])).toBeLessThanOrEqual(1); + } + }); +}); + +describe('smoothToward', () => { + it('reaches ~95% of the way in 3τ and jumps with τ 0', () => { + let v = 0; + for (let t = 0; t < 1.5; t += 0.25) v = smoothToward(v, 1, 0.25, 0.5); + expect(v).toBeGreaterThan(0.94); + expect(v).toBeLessThan(1); + expect(smoothToward(0.2, 0.9, 0.1, 0)).toBe(0.9); + expect(smoothToward(0.2, 0.9, 0, 1)).toBe(0.2); + }); +}); diff --git a/apps/studio/src/audio/mix.ts b/apps/studio/src/audio/mix.ts new file mode 100644 index 0000000..8e627a3 --- /dev/null +++ b/apps/studio/src/audio/mix.ts @@ -0,0 +1,113 @@ +/** + * The ambient bed's mix rules (contract §4b), pure: from where the camera is + * (the town's listener) to each layer's level. The engine smooths these and + * turns them into gains and events. + * + * day = 1 − night, z = zoom01 (0 the whole town … 1 street level), green = + * greenery near the camera. Murmur comes only from `murmur` sources, which + * the town derives from simulated occupancy at busy places; ids in `exclude` + * (agent buildings and lots) are dropped as a second guard. + */ +import type { AudioListener } from './types'; + +export interface AmbientMix { + /** Layer levels 0…1, before each layer's trim and the ambient volume. */ + breeze: number; + /** High wind: the wide view's air. */ + wind: number; + /** Bird calls per second (day). */ + birdsRate: number; + /** Night insects. */ + crickets: number; + murmur: number; + murmurPan: number; + water: number; + waterPan: number; + traffic: number; + trafficPan: number; + /** 0 … 1 day, for traffic pass-bys. */ + day: number; + /** Inside a building: the outdoor bed −9 dB through a 700 Hz lowpass, plus room tone. */ + interior: boolean; + room: number; +} + +export const SILENT_MIX: Readonly = Object.freeze({ + breeze: 0, + wind: 0, + birdsRate: 0, + crickets: 0, + murmur: 0, + murmurPan: 0, + water: 0, + waterPan: 0, + traffic: 0, + trafficPan: 0, + day: 1, + interior: false, + room: 0, +}); + +/** The outdoor bed inside a building: −9 dB. */ +export const INTERIOR_OUTDOOR_GAIN = Math.pow(10, -9 / 20); +export const ROOM_TONE = 0.08; +/** A small town always has a few birds, even over rooftops. */ +const GREEN_FLOOR = 0.1; + +const clamp01 = (v: number) => (Number.isFinite(v) ? Math.min(1, Math.max(0, v)) : 0); +const clampPan = (v: number) => (Number.isFinite(v) ? Math.min(1, Math.max(-1, v)) : 0); + +export function ambientMix(l: AudioListener | null, opts: { reduce?: boolean; exclude?: ReadonlySet } = {}): AmbientMix { + if (!l || opts.reduce) return { ...SILENT_MIX }; + const z = clamp01(l.zoom01); + const night = clamp01(l.night); + const day = 1 - night; + const sources = opts.exclude?.size ? l.sources.filter((s) => !opts.exclude!.has(s.id)) : l.sources; + + let green = GREEN_FLOOR; + let murmurSum = 0; + let murmurPanSum = 0; + let water = 0; + let waterPan = 0; + let road = 0; + let trafficPan = 0; + for (const s of sources) { + const g = clamp01(s.gain); + if (s.kind === 'green') green += g; + else if (s.kind === 'field') green += 0.5 * g; + else if (s.kind === 'murmur') { + murmurSum += g; + murmurPanSum += g * clampPan(s.pan); + } else if (s.kind === 'water' && g > water) { + water = g; + waterPan = clampPan(s.pan); + } else if (s.kind === 'road' && g > road) { + road = g; + trafficPan = clampPan(s.pan); + } + } + green = clamp01(green); + const live = l.paused ? 0 : 1; + const interior = l.entered !== null || l.level === 'interior'; + return { + breeze: (0.35 + 0.25 * (1 - z)) * (l.paused ? 0.5 : 1), + wind: 0.25 * (1 - z) * (1 - z), + birdsRate: 0.5 * day * green * (0.25 + 0.75 * z) * live, + crickets: night * (0.35 + 0.65 * green) * live, + murmur: Math.min(1, murmurSum) * (0.2 + 0.8 * z) * live, + murmurPan: murmurSum > 0 ? clampPan(murmurPanSum / murmurSum) : 0, + water: water * z, + waterPan, + traffic: (0.15 + 0.2 * (1 - z)) * (0.4 + 0.6 * day) * live, + trafficPan, + day, + interior, + room: interior ? ROOM_TONE : 0, + }; +} + +/** One step of exponential smoothing toward `target` (τ in seconds). τ ≤ 0 jumps. */ +export function smoothToward(current: number, target: number, dt: number, tau: number): number { + if (tau <= 0 || dt <= 0) return tau <= 0 ? target : current; + return current + (target - current) * (1 - Math.exp(-dt / tau)); +} diff --git a/apps/studio/src/audio/react.tsx b/apps/studio/src/audio/react.tsx new file mode 100644 index 0000000..deb65a0 --- /dev/null +++ b/apps/studio/src/audio/react.tsx @@ -0,0 +1,219 @@ +'use client'; +/** + * Sound in the HUD: the speaker toggle (clock widget) and the Sound tab of + * the Esc menu. Both read the engine's settings with useSyncExternalStore + * (the server renders the defaults, so hydration matches). + */ +import { useId, useSyncExternalStore, type CSSProperties, type ReactNode } from 'react'; +import { Button } from '../ui/Button'; +import { SegmentedControl } from '../ui/SegmentedControl'; +import styles from './audio.module.css'; +import { getAudio, getAudioEngine } from './engine'; +import { DEFAULT_AUDIO_SETTINGS } from './settings'; +import type { AudioSettings, AudioState } from './types'; + +const serverSettings = () => DEFAULT_AUDIO_SETTINGS; +const serverState = (): AudioState => 'unsupported'; + +export function useAudioSettings(): [AudioSettings, (p: Partial) => void] { + const audio = getAudio(); + const settings = useSyncExternalStore(audio.subscribe, audio.getSettings, serverSettings); + return [settings, audio.update]; +} + +export function useAudioState(): AudioState { + const audio = getAudio(); + return useSyncExternalStore( + audio.subscribe, + () => audio.state, + serverState, + ); +} + +function SpeakerIcon({ on }: { on: boolean }) { + return ( + + ); +} + +function toggleTitle(on: boolean, state: AudioState): string { + if (state === 'unsupported') return 'Sound is not available in this browser'; + if (!on) return 'Sound off. Click to turn it on.'; + if (state === 'locked') return 'Sound on: it starts with your first click or key press. Click to mute.'; + return 'Sound on. Click to mute.'; +} + +/** The HUD speaker toggle: icon + aria-pressed; the visible text says on or off (not by colour alone). */ +export function SoundToggle({ compact = false }: { compact?: boolean }) { + const [settings, update] = useAudioSettings(); + const state = useAudioState(); + const on = settings.enabled; + const unsupported = state === 'unsupported'; + return ( + + ); +} + +function Slider({ label, help, value, disabled, onChange, extra }: { label: string; help?: string; value: number; disabled?: boolean; onChange: (v: number) => void; extra?: ReactNode }) { + const id = useId(); + const pct = Math.round(value * 100); + return ( +
+
+ + {help &&

{help}

} +
+
+ onChange(Number(e.currentTarget.value) / 100)} + /> + + {pct === 0 ? 'Off' : `${pct}%`} + + {extra ??
+
+ ); +} + +function stateLine(state: AudioState, on: boolean): string { + if (state === 'unsupported') return 'Sound is not available in this browser (no Web Audio).'; + if (!on) return 'Sound is off.'; + if (state === 'locked') return 'Sound starts with your first click or key press in the world.'; + if (state === 'suspended') return 'Sound is paused (hidden tab, or nothing to hear right now).'; + return 'Sound is playing.'; +} + +/** The Esc menu's Sound tab: on/off, master / ambient / UI / events volumes, reduce sound. Saved per viewer. */ +export function SoundSettings() { + const [s, update] = useAudioSettings(); + const state = useAudioState(); + const unsupported = state === 'unsupported'; + const preview = (what: 'ui' | 'events') => getAudioEngine()?.preview(what); + return ( +
+

+ {stateLine(state, s.enabled)} +

+
+
+

Sound

+

Also the speaker button next to the clock.

+
+
+ { + update({ enabled: v === 'on' }); + getAudio().unlock(); + }} + /> +
+
+ update({ master: v })} /> + update({ ambient: v })} + /> + update({ ui: v })} + extra={ + + } + /> + update({ events: v })} + extra={ + + } + /> +
+
+

Reduce sound

+

No ambient bed or hover ticks, quieter interface, and only the needs-approval, incident and task-done cues.

+
+
+ update({ reduce: v === 'on' })} + /> +
+
+

+ Every event sound repeats a notification you can read: a sound never stands alone. Sounds are made in your browser (no audio files), + pause while this tab is hidden, and these settings are saved in this browser only. +

+
+ ); +} diff --git a/apps/studio/src/audio/render.ts b/apps/studio/src/audio/render.ts new file mode 100644 index 0000000..0170ab2 --- /dev/null +++ b/apps/studio/src/audio/render.ts @@ -0,0 +1,258 @@ +/** + * Offline renders (browser only): every sound and a few ambient scenes are + * rendered with an OfflineAudioContext through the real graph, soundscape and + * voices, then measured (peak and RMS). `check/render-check.mjs` runs this in + * headless Chromium and fails when a sound is silent, clips (peak ≥ 0.9), or + * the bed is not quiet, or when a setting does not change the output. + */ +import { LOOKAHEAD, Soundscape } from './ambient'; +import { buildGraph, busTargets, rampTo, type AudioGraph } from './graph'; +import { SILENT_MIX, ambientMix, type AmbientMix } from './mix'; +import { DEFAULT_AUDIO_SETTINGS } from './settings'; +import { birdVoice, cueVoice, makeBuffers, seededRng, uiVoice, whooshVoice, type Synth } from './synth'; +import { EVENT_CUES, UI_SOUNDS, type AudioListener, type AudioSettings, type EventCue, type UiSound } from './types'; + +export const SAMPLE_RATE = 48000; +export const CLIP = 0.9; + +export interface Measure { + name: string; + peak: number; + rms: number; + peakDb: number; + rmsDb: number; +} + +const db = (v: number) => (v > 0 ? Math.round(20 * Math.log10(v) * 10) / 10 : -Infinity); + +function measure(name: string, buf: AudioBuffer, from = 0, to = buf.duration): Measure { + let peak = 0; + let sum = 0; + let n = 0; + const a = Math.floor(from * buf.sampleRate); + const b = Math.min(buf.length, Math.floor(to * buf.sampleRate)); + for (let ch = 0; ch < buf.numberOfChannels; ch++) { + const d = buf.getChannelData(ch); + for (let i = 0; i < buf.length; i++) peak = Math.max(peak, Math.abs(d[i]!)); + for (let i = a; i < b; i++) { + sum += d[i]! * d[i]!; + n++; + } + } + const rms = Math.sqrt(sum / Math.max(1, n)); + return { name, peak, rms, peakDb: db(peak), rmsDb: db(rms) }; +} + +interface Scene { + ctx: OfflineAudioContext; + graph: AudioGraph; + synth: Synth; +} + +async function render(seconds: number, settings: AudioSettings, play: (s: Scene) => void, opts: { interior?: boolean; seed?: number } = {}): Promise { + const ctx = new OfflineAudioContext(2, Math.round(seconds * SAMPLE_RATE), SAMPLE_RATE); + const rng = seededRng(opts.seed ?? 42); + const buffers = makeBuffers(ctx, rng); + const graph = buildGraph(ctx, buffers); + const tg = busTargets(settings, { audible: true, interior: opts.interior ?? false }); + rampTo(graph.master.gain, tg.master, 0, 0); + rampTo(graph.ambient.gain, tg.ambient, 0, 0); + rampTo(graph.birdsSend.gain, tg.birdsSend, 0, 0); + rampTo(graph.ui.gain, tg.ui, 0, 0); + rampTo(graph.events.gain, tg.events, 0, 0); + play({ ctx, graph, synth: { ctx, buffers, rng } }); + return ctx.startRendering(); +} + +/** Run the soundscape for `seconds` of a listener (its timer, pumped ahead of the render). */ +function bed(s: Scene, listener: AudioListener, seconds: number, settings: AudioSettings, only?: (m: AmbientMix) => AmbientMix): Soundscape { + const scape = new Soundscape(s.graph, s.synth.rng, (t, species, pan, peak) => { + birdVoice(s.synth, s.graph.birds, t, species, pan, peak); + }); + scape.start(0); + const full = ambientMix(listener, { reduce: settings.reduce }); + const mix = only ? only(full) : full; + let last = 0; + for (let now = 0; now < seconds - LOOKAHEAD; now += 0.25) { + scape.update(now, mix, now - last); + last = now; + } + return scape; +} + +export const SCENES: Record = { + 'town · day': { + x: 48, z: 36, zoom01: 0, az: 45, entered: null, level: 'town', minute: 11 * 60, night: 0, paused: false, + sources: [ + { kind: 'green', id: 'forest', gain: 0.5, pan: -0.2 }, + { kind: 'field', id: 'field_1', gain: 0.4, pan: 0.5 }, + { kind: 'road', id: 'westway', gain: 0.5, pan: -0.6 }, + { kind: 'water', id: 'brook', gain: 0.3, pan: 0.2 }, + { kind: 'murmur', id: 'spot_fountain_rim', gain: 0.3, pan: 0 }, + ], + }, + 'street · café · day': { + x: 40, z: 30, zoom01: 1, az: 45, entered: null, level: 'street', minute: 13 * 60, night: 0, paused: false, + sources: [ + { kind: 'murmur', id: 'spot_cafe_patio', gain: 0.9, pan: -0.3 }, + { kind: 'murmur', id: 'bld_corner_cafe', gain: 0.6, pan: -0.5 }, + { kind: 'water', id: 'lm_fountain', gain: 0.7, pan: 0.4 }, + { kind: 'green', id: 'park_commons', gain: 0.4, pan: 0.6 }, + { kind: 'road', id: 'brook_road', gain: 0.3, pan: 0.8 }, + ], + }, + 'district · night': { + x: 60, z: 40, zoom01: 0.5, az: 200, entered: null, level: 'district', minute: 23 * 60, night: 1, paused: false, + sources: [ + { kind: 'green', id: 'park_commons', gain: 0.6, pan: -0.4 }, + { kind: 'field', id: 'field_2', gain: 0.3, pan: 0.7 }, + { kind: 'road', id: 'brook_road', gain: 0.3, pan: 0.1 }, + ], + }, + 'inside the café': { + x: 40, z: 30, zoom01: 1, az: 45, entered: 'bld_corner_cafe', level: 'interior', minute: 13 * 60, night: 0, paused: false, + sources: [ + { kind: 'murmur', id: 'spot_cafe_patio', gain: 0.9, pan: -0.3 }, + { kind: 'green', id: 'park_commons', gain: 0.4, pan: 0.6 }, + ], + }, +}; +SCENES['town · day · paused'] = { ...SCENES['town · day']!, paused: true }; + +const MAX: AudioSettings = { enabled: true, master: 1, ambient: 1, ui: 1, events: 1, reduce: false }; +const BED_SECONDS = 10; +const BED_FROM = 3; + +export async function renderUi(sound: UiSound | 'whoosh', settings: AudioSettings = DEFAULT_AUDIO_SETTINGS): Promise { + const buf = await render(1.6, settings, (s) => { + if (sound === 'whoosh') whooshVoice(s.synth, s.graph.ui, 0.05, 0.7, 0.6); + else uiVoice(s.synth, s.graph.ui, 0.05, sound); + }); + return measure(sound, buf); +} + +export async function renderCue(cue: EventCue, settings: AudioSettings = DEFAULT_AUDIO_SETTINGS): Promise { + const buf = await render(2.5, settings, (s) => void cueVoice(s.synth, s.graph.events, 0.05, cue, 0)); + return measure(cue, buf); +} + +export async function renderBed(name: string, settings: AudioSettings = DEFAULT_AUDIO_SETTINGS, only?: (m: AmbientMix) => AmbientMix): Promise { + const l = SCENES[name]!; + const buf = await render(BED_SECONDS, settings, (s) => void bed(s, l, BED_SECONDS, settings, only), { interior: l.entered !== null }); + return measure(name, buf, BED_FROM); +} + +const LAYERS = { + breeze: ['breeze'], + wind: ['wind'], + birds: ['birdsRate'], + crickets: ['crickets'], + murmur: ['murmur', 'murmurPan'], + water: ['water', 'waterPan'], + traffic: ['traffic', 'trafficPan'], + room: ['room'], +} as const satisfies Record; + +/** Each layer of a scene alone (for tuning the trims). */ +export async function renderLayers(name: string, settings: AudioSettings = DEFAULT_AUDIO_SETTINGS): Promise { + const out: Measure[] = []; + for (const [layer, keys] of Object.entries(LAYERS)) { + const m = await renderBed(name, settings, (full) => { + const one: AmbientMix = { ...SILENT_MIX, day: full.day, interior: full.interior }; + for (const k of keys) (one as unknown as Record)[k] = full[k] as number; + return one; + }); + out.push({ ...m, name: `${name} · ${layer}` }); + } + return out; +} + +export interface CheckRow { + group: 'ui' | 'cue' | 'bed' | 'settings' | 'stack'; + name: string; + measure?: Measure; + expect: string; + ok: boolean; +} + +/** Every check (S4 and the brief): non-silent, never clipping, a quiet bed, settings that change the output. */ +export async function runRenderCheck(): Promise<{ ok: boolean; rows: CheckRow[] }> { + const rows: CheckRow[] = []; + const add = (r: CheckRow) => rows.push(r); + const SILENT_DB = -90; + + for (const sound of [...UI_SOUNDS, 'whoosh' as const]) { + const d = await renderUi(sound); + const m = await renderUi(sound, MAX); + add({ group: 'ui', name: `${sound} (default)`, measure: d, expect: 'peak > −50 dBFS and < 0.9', ok: d.peakDb > -50 && d.peak < CLIP }); + add({ group: 'ui', name: `${sound} (max volume)`, measure: m, expect: 'peak < 0.9', ok: m.peak < CLIP && m.peakDb > -40 }); + } + for (const cue of EVENT_CUES) { + const d = await renderCue(cue); + const m = await renderCue(cue, MAX); + add({ group: 'cue', name: `${cue} (default)`, measure: d, expect: 'peak > −40 dBFS and < 0.9', ok: d.peakDb > -40 && d.peak < CLIP }); + add({ group: 'cue', name: `${cue} (max volume)`, measure: m, expect: 'peak < 0.9', ok: m.peak < CLIP }); + } + for (const name of Object.keys(SCENES)) { + const d = await renderBed(name); + add({ group: 'bed', name: `${name} (default)`, measure: d, expect: 'RMS −56…−32 dBFS, peak < 0.9', ok: d.rmsDb > -56 && d.rmsDb < -32 && d.peak < CLIP }); + } + const loud = await renderBed('street · café · day', MAX); + add({ group: 'bed', name: 'street · café · day (max volume)', measure: loud, expect: 'RMS < −12 dBFS, peak < 0.9', ok: loud.rmsDb < -12 && loud.peak < CLIP }); + + // The busiest moment the pacing allows, at full volume: the busiest bed, the four cues 1.2 s apart + // (the limiter's spacing), clicks and a camera whoosh on top of the first. The worst of five seeds. + let stack: Measure | null = null; + for (const seed of [1, 7, 42, 99, 2024]) { + const buf = await render( + 9, + MAX, + (s) => { + bed(s, SCENES['street · café · day']!, 9, MAX); + EVENT_CUES.forEach((cue, i) => cueVoice(s.synth, s.graph.events, 3 + i * 1.2, cue, 0)); + uiVoice(s.synth, s.graph.ui, 3, 'select'); + uiVoice(s.synth, s.graph.ui, 3.02, 'menu'); + uiVoice(s.synth, s.graph.ui, 4.2, 'error'); + whooshVoice(s.synth, s.graph.ui, 2.8, 0.8, 0.5); + }, + { seed }, + ); + const m = measure('busiest moment (max volume, worst of 5)', buf); + if (!stack || m.peak > stack.peak) stack = m; + } + stack = stack!; + add({ group: 'stack', name: stack.name, measure: stack, expect: 'peak < 0.9', ok: stack.peak < CLIP }); + + // Settings audibly change the output. + const base = await renderBed('street · café · day'); + const variants: [string, Partial, (m: Measure) => boolean, string][] = [ + ['bed · master 30%', { master: 0.3 }, (m) => m.rmsDb < base.rmsDb - 6, '≥ 6 dB quieter'], + ['bed · ambient 100%', { ambient: 1 }, (m) => m.rmsDb > base.rmsDb + 6, '≥ 6 dB louder'], + ['bed · ambient off', { ambient: 0 }, (m) => m.rmsDb < SILENT_DB, 'silent'], + ['bed · sound off (mute)', { enabled: false }, (m) => m.rmsDb < SILENT_DB, 'silent'], + ['bed · reduce sound', { reduce: true }, (m) => m.rmsDb < SILENT_DB, 'silent'], + ]; + for (const [name, p, ok, expect] of variants) { + const m = await renderBed('street · café · day', { ...DEFAULT_AUDIO_SETTINGS, ...p }); + add({ group: 'settings', name, measure: m, expect: `${expect} (default ${base.rmsDb} dB)`, ok: ok(m) }); + } + const sel = await renderUi('select'); + const uiChecks: [string, Partial, (m: Measure) => boolean, string][] = [ + ['select · UI off', { ui: 0 }, (m) => m.rmsDb < SILENT_DB, 'silent'], + ['select · UI 100%', { ui: 1 }, (m) => m.rmsDb > sel.rmsDb + 6, '≥ 6 dB louder'], + ['select · reduce sound (−6 dB)', { reduce: true }, (m) => Math.abs(m.rmsDb - (sel.rmsDb - 6)) < 1, '6 dB quieter'], + ]; + for (const [name, p, ok, expect] of uiChecks) { + const m = await renderUi('select', { ...DEFAULT_AUDIO_SETTINGS, ...p }); + add({ group: 'settings', name, measure: m, expect: `${expect} (default ${sel.rmsDb} dB)`, ok: ok(m) }); + } + const done = await renderCue('taskDone'); + for (const [name, p, ok, expect] of [ + ['taskDone · events off', { events: 0 }, (m: Measure) => m.rmsDb < SILENT_DB, 'silent'], + ['taskDone · events 30%', { events: 0.3 }, (m: Measure) => m.rmsDb < done.rmsDb - 6, '≥ 6 dB quieter'], + ] as const) { + const m = await renderCue('taskDone', { ...DEFAULT_AUDIO_SETTINGS, ...p }); + add({ group: 'settings', name, measure: m, expect: `${expect} (default ${done.rmsDb} dB)`, ok: ok(m) }); + } + return { ok: rows.every((r) => r.ok), rows }; +} diff --git a/apps/studio/src/audio/settings.test.ts b/apps/studio/src/audio/settings.test.ts new file mode 100644 index 0000000..32bd986 --- /dev/null +++ b/apps/studio/src/audio/settings.test.ts @@ -0,0 +1,59 @@ +import { describe, expect, it } from 'vitest'; +import { AUDIO_SETTINGS_KEY, DEFAULT_AUDIO_SETTINGS, loadAudioSettings, mergeAudioSettings, parseAudioSettings, saveAudioSettings, sliderGain } from './settings'; + +function memoryStorage() { + const m = new Map(); + return { getItem: (k: string) => m.get(k) ?? null, setItem: (k: string, v: string) => void m.set(k, v), map: m }; +} +const throwing = { + getItem: () => { + throw new Error('SecurityError'); + }, + setItem: () => { + throw new Error('QuotaExceededError'); + }, +}; + +describe('audio settings', () => { + it('defaults: on, master 0.6, ambient 0.5, UI 0.4, events 0.7, not reduced', () => { + expect(DEFAULT_AUDIO_SETTINGS).toEqual({ enabled: true, master: 0.6, ambient: 0.5, ui: 0.4, events: 0.7, reduce: false }); + expect(loadAudioSettings(null)).toEqual(DEFAULT_AUDIO_SETTINGS); + }); + + it('clamps volumes, ignores junk and unknown fields', () => { + expect(parseAudioSettings({ master: 2, ambient: -1, ui: Number.NaN, events: '0.3', reduce: 'yes', enabled: false, extra: 1 })).toEqual({ + ...DEFAULT_AUDIO_SETTINGS, + master: 1, + ambient: 0, + enabled: false, + }); + expect(parseAudioSettings(null)).toEqual(DEFAULT_AUDIO_SETTINGS); + expect(parseAudioSettings([1, 2])).toEqual(DEFAULT_AUDIO_SETTINGS); + expect(mergeAudioSettings({ ...DEFAULT_AUDIO_SETTINGS, ui: 0.9 }, { master: 0.333 })).toEqual({ ...DEFAULT_AUDIO_SETTINGS, ui: 0.9, master: 0.33 }); + }); + + it('persists per viewer in runloop.audio.v1 and reads it back', () => { + const st = memoryStorage(); + const s = { ...DEFAULT_AUDIO_SETTINGS, master: 0.25, reduce: true, enabled: false }; + expect(saveAudioSettings(s, st)).toBe(true); + expect(JSON.parse(st.map.get(AUDIO_SETTINGS_KEY)!)).toEqual(s); + expect(loadAudioSettings(st)).toEqual(s); + }); + + it('survives storage that throws or holds corrupt data', () => { + expect(loadAudioSettings(throwing)).toEqual(DEFAULT_AUDIO_SETTINGS); + expect(saveAudioSettings(DEFAULT_AUDIO_SETTINGS, throwing)).toBe(false); + const st = memoryStorage(); + st.map.set(AUDIO_SETTINGS_KEY, '{not json'); + expect(loadAudioSettings(st)).toEqual(DEFAULT_AUDIO_SETTINGS); + st.map.set(AUDIO_SETTINGS_KEY, JSON.stringify({ ui: 0.1 })); + expect(loadAudioSettings(st)).toEqual({ ...DEFAULT_AUDIO_SETTINGS, ui: 0.1 }); + }); + + it('slider taper: 0 is silent, half is −12 dB, full is unity', () => { + expect(sliderGain(0)).toBe(0); + expect(20 * Math.log10(sliderGain(0.5))).toBeCloseTo(-12.04, 1); + expect(sliderGain(1)).toBe(1); + expect(sliderGain(3)).toBe(1); + }); +}); diff --git a/apps/studio/src/audio/settings.ts b/apps/studio/src/audio/settings.ts new file mode 100644 index 0000000..8fe01a0 --- /dev/null +++ b/apps/studio/src/audio/settings.ts @@ -0,0 +1,91 @@ +/** + * Sound settings: defaults, parsing (every field clamped, anything unreadable + * falls back to its default), and per-viewer persistence in localStorage + * `runloop.audio.v1`. Storage can be missing or throw (private windows, + * blocked site data): reads fall back to the defaults, writes are skipped. + */ +import type { AudioSettings } from './types'; + +export const AUDIO_SETTINGS_KEY = 'runloop.audio.v1'; + +export const DEFAULT_AUDIO_SETTINGS: Readonly = Object.freeze({ + enabled: true, + master: 0.6, + ambient: 0.5, + ui: 0.4, + events: 0.7, + reduce: false, +}); + +const VOLUMES = ['master', 'ambient', 'ui', 'events'] as const; +const FLAGS = ['enabled', 'reduce'] as const; + +const clamp01 = (v: number) => Math.min(1, Math.max(0, v)); + +/** A complete, clamped settings object from anything (unknown fields are dropped). */ +export function parseAudioSettings(raw: unknown, base: AudioSettings = DEFAULT_AUDIO_SETTINGS): AudioSettings { + const out: AudioSettings = { ...base }; + if (!raw || typeof raw !== 'object') return out; + const r = raw as Record; + for (const k of VOLUMES) { + const v = r[k]; + if (typeof v === 'number' && Number.isFinite(v)) out[k] = Math.round(clamp01(v) * 100) / 100; + } + for (const k of FLAGS) if (typeof r[k] === 'boolean') out[k] = r[k] as boolean; + return out; +} + +/** Merge a partial update into settings (clamped). */ +export function mergeAudioSettings(s: AudioSettings, p: Partial): AudioSettings { + return parseAudioSettings(p, s); +} + +export function sameAudioSettings(a: AudioSettings, b: AudioSettings): boolean { + return VOLUMES.every((k) => a[k] === b[k]) && FLAGS.every((k) => a[k] === b[k]); +} + +type ReadStorage = Pick; +type WriteStorage = Pick; + +function defaultStorage(): Storage | null { + try { + return typeof window === 'undefined' ? null : window.localStorage; + } catch { + return null; + } +} + +export function loadAudioSettings(storage: ReadStorage | null = defaultStorage()): AudioSettings { + if (!storage) return { ...DEFAULT_AUDIO_SETTINGS }; + try { + const raw = storage.getItem(AUDIO_SETTINGS_KEY); + return raw ? parseAudioSettings(JSON.parse(raw) as unknown) : { ...DEFAULT_AUDIO_SETTINGS }; + } catch { + return { ...DEFAULT_AUDIO_SETTINGS }; + } +} + +/** Returns false when the settings could not be stored (they still apply for this page). */ +export function saveAudioSettings(s: AudioSettings, storage: WriteStorage | null = defaultStorage()): boolean { + if (!storage) return false; + try { + storage.setItem(AUDIO_SETTINGS_KEY, JSON.stringify(parseAudioSettings(s))); + return true; + } catch { + return false; + } +} + +/** + * A slider (0…1) to a gain. Squared, a rough loudness taper: half-way is + * −12 dB, so the low half of the slider stays usable for a quiet bed. + */ +export function sliderGain(v: number): number { + const c = clamp01(v); + return c * c; +} + +/** dB of a linear gain (−Infinity for 0). */ +export function toDb(g: number): number { + return g > 0 ? 20 * Math.log10(g) : -Infinity; +} diff --git a/apps/studio/src/audio/synth.ts b/apps/studio/src/audio/synth.ts new file mode 100644 index 0000000..a0c3cae --- /dev/null +++ b/apps/studio/src/audio/synth.ts @@ -0,0 +1,386 @@ +/** + * Sound synthesis (contract §4b): everything is generated with the Web Audio + * API, with no audio files, samples or licences. Noise buffers and the reverb impulse + * are made once per context; each UI sound, whoosh, cue and bird call is a + * short-lived voice of oscillators and filtered noise with its own envelope. + * + * Levels here are pre-bus peaks: at full volume sliders a voice peaks at + * about these values; the buses and the master scale them down (see + * `render.ts` for the measured output). + */ +import type { EventCue, UiSound } from './types'; + +export type Rng = () => number; + +/** A deterministic PRNG (mulberry32), so renders and tests repeat. */ +export function seededRng(seed = 1): Rng { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +export const between = (rng: Rng, lo: number, hi: number) => lo + (hi - lo) * rng(); + +export interface Buffers { + /** 4 s of looping pink noise (RMS ≈ 0.25). */ + pink: AudioBuffer; + /** 4 s of looping brown noise (RMS ≈ 0.25). */ + brown: AudioBuffer; + /** A 1.6 s stereo room: decaying, darkening noise. */ + impulse: AudioBuffer; +} + +const NOISE_SECONDS = 4; +const LOOP_FADE_SECONDS = 0.05; + +/** Make a loop seamless: the last `fade` samples blend into the first (equal power). */ +function loopable(raw: Float32Array, n: number, fade: number): Float32Array { + const out = raw.slice(0, n); + for (let i = 0; i < fade; i++) { + const a = Math.sqrt(i / fade); + out[i] = raw[i]! * a + raw[n + i]! * Math.sqrt(1 - a * a); + } + return out; +} + +function normalise(x: Float32Array, rms: number): void { + let mean = 0; + for (const v of x) mean += v; + mean /= x.length; + let sum = 0; + for (let i = 0; i < x.length; i++) { + x[i] = x[i]! - mean; + sum += x[i]! * x[i]!; + } + const k = rms / Math.sqrt(sum / x.length || 1); + for (let i = 0; i < x.length; i++) x[i] = Math.max(-0.98, Math.min(0.98, x[i]! * k)); +} + +/** Pink noise (Paul Kellet's filter). */ +export function pinkNoise(len: number, rng: Rng): Float32Array { + const out = new Float32Array(len); + let b0 = 0, b1 = 0, b2 = 0, b3 = 0, b4 = 0, b5 = 0, b6 = 0; + for (let i = 0; i < len; i++) { + const w = rng() * 2 - 1; + b0 = 0.99886 * b0 + w * 0.0555179; + b1 = 0.99332 * b1 + w * 0.0750759; + b2 = 0.969 * b2 + w * 0.153852; + b3 = 0.8665 * b3 + w * 0.3104856; + b4 = 0.55 * b4 + w * 0.5329522; + b5 = -0.7616 * b5 - w * 0.016898; + out[i] = b0 + b1 + b2 + b3 + b4 + b5 + b6 + w * 0.5362; + b6 = w * 0.115926; + } + return out; +} + +/** Brown noise (leaky integrated white noise, DC removed). */ +export function brownNoise(len: number, rng: Rng): Float32Array { + const out = new Float32Array(len); + let last = 0; + let px = 0; + let py = 0; + for (let i = 0; i < len; i++) { + last = (last + 0.02 * (rng() * 2 - 1)) / 1.02; + // DC blocker + py = last - px + 0.999 * py; + px = last; + out[i] = py; + } + return out; +} + +export function makeBuffers(ctx: BaseAudioContext, rng: Rng): Buffers { + const sr = ctx.sampleRate; + const n = Math.round(NOISE_SECONDS * sr); + const fade = Math.round(LOOP_FADE_SECONDS * sr); + const make = (gen: (len: number, rng: Rng) => Float32Array) => { + const raw = gen(n + fade, rng); + normalise(raw, 0.25); + const data = loopable(raw, n, fade); + const b = ctx.createBuffer(1, n, sr); + b.getChannelData(0).set(data); + return b; + }; + const pink = make(pinkNoise); + const brown = make(brownNoise); + + const len = Math.round(1.6 * sr); + const impulse = ctx.createBuffer(2, len, sr); + for (let ch = 0; ch < 2; ch++) { + const d = impulse.getChannelData(ch); + let lp = 0; + for (let i = 0; i < len; i++) { + const t = i / sr; + const env = Math.exp(-t / 0.23) * Math.min(1, t / 0.01); + // The tail gets darker: a one-pole lowpass closing over time. + const k = 0.15 + 0.7 * Math.min(1, t / 1.2); + lp = lp * k + (rng() * 2 - 1) * (1 - k); + d[i] = lp * env; + } + } + return { pink, brown, impulse }; +} + +// ── Voices ─────────────────────────────────────────────────────────── + +export type VoiceKind = 'ui' | 'bird' | 'event'; + +/** Each kind's level over the ambient bed (calibrated with `render.ts`): UI +2 dB, event cues +3 dB. */ +export const VOICE_LEVEL: Readonly> = { ui: 1.26, bird: 1, event: 1.41 }; + +/** A one-shot sound: its sources stop by `end`; `out` is its last gain (faded to steal it). */ +export interface Voice { + kind: VoiceKind; + start: number; + end: number; + out: GainNode; + sources: AudioScheduledSourceNode[]; +} + +export interface Synth { + ctx: BaseAudioContext; + buffers: Buffers; + rng: Rng; +} + +/** A stereo panner, or a plain gain where StereoPannerNode is missing. */ +export function createPan(ctx: BaseAudioContext, pan = 0): { node: AudioNode; pan: AudioParam | null } { + if (typeof ctx.createStereoPanner === 'function') { + const p = ctx.createStereoPanner(); + p.pan.value = Math.max(-1, Math.min(1, pan)); + return { node: p, pan: p.pan }; + } + return { node: ctx.createGain(), pan: null }; +} + +class VoiceBuilder { + readonly out: GainNode; + readonly sources: AudioScheduledSourceNode[] = []; + end: number; + constructor( + readonly s: Synth, + dest: AudioNode, + readonly t: number, + readonly kind: VoiceKind, + ) { + this.out = s.ctx.createGain(); + this.out.gain.value = VOICE_LEVEL[kind]; + this.out.connect(dest); + this.end = t; + } + + /** An oscillator with an attack/decay envelope; `to` glides the pitch (exponentially) over `glide` s. */ + tone(o: { type?: OscillatorType; freq: number; to?: number; glide?: number; at?: number; attack: number; decay: number; peak: number; via?: AudioNode }): OscillatorNode { + const { ctx } = this.s; + const t = this.t + (o.at ?? 0); + const osc = ctx.createOscillator(); + osc.type = o.type ?? 'sine'; + osc.frequency.setValueAtTime(o.freq, t); + if (o.to) osc.frequency.exponentialRampToValueAtTime(o.to, t + (o.glide ?? 0.09)); + const g = ctx.createGain(); + envelope(g.gain, t, o.attack, o.peak, o.decay); + osc.connect(g).connect(o.via ?? this.out); + this.add(osc, t, t + o.attack + o.decay * 1.6 + 0.02); + return osc; + } + + /** A burst of filtered noise. */ + noise(o: { buffer?: AudioBuffer; filter: BiquadFilterType; freq: number; q?: number; at?: number; attack: number; decay: number; peak: number }): void { + const { ctx, buffers, rng } = this.s; + const t = this.t + (o.at ?? 0); + const buffer = o.buffer ?? buffers.pink; + const src = ctx.createBufferSource(); + src.buffer = buffer; + const f = ctx.createBiquadFilter(); + f.type = o.filter; + f.frequency.value = o.freq; + f.Q.value = o.q ?? 0.7; + const g = ctx.createGain(); + envelope(g.gain, t, o.attack, o.peak, o.decay); + src.connect(f).connect(g).connect(this.out); + const end = t + o.attack + o.decay * 1.6 + 0.02; + src.start(t, between(rng, 0, Math.max(0, buffer.duration - (end - t) - 0.1))); + this.add(src, t, end, true); + } + + /** Track a source: start it (unless `started`, e.g. a buffer started at an offset) and stop it at `end`. */ + add(src: AudioScheduledSourceNode, start: number, end: number, started = false): void { + if (!started) src.start(start); + src.stop(end); + this.sources.push(src); + this.end = Math.max(this.end, end); + } + + done(): Voice { + const last = this.sources[this.sources.length - 1]; + const out = this.out; + if (last) last.onended = () => out.disconnect(); + return { kind: this.kind, start: this.t, end: this.end, out, sources: this.sources }; + } +} + +/** Attack (linear) then an exponential decay to ~2% after `decay` seconds. */ +export function envelope(p: AudioParam, t: number, attack: number, peak: number, decay: number): void { + p.setValueAtTime(0, t); + p.linearRampToValueAtTime(peak, t + attack); + p.setTargetAtTime(0, t + attack, decay / 4); +} + +const C5 = 523.25, E5 = 659.25, G5 = 783.99, C6 = 1046.5, A5 = 880, A4 = 440, F4 = 349.23, G4 = 392, E4 = 329.63; + +/** The interface sounds: short, soft, mostly sine. */ +export function uiVoice(s: Synth, dest: AudioNode, t: number, sound: UiSound): Voice { + const v = new VoiceBuilder(s, dest, t, 'ui'); + switch (sound) { + case 'hover': + v.tone({ freq: 2600, attack: 0.001, decay: 0.025, peak: 0.14 }); + break; + case 'select': + v.tone({ type: 'triangle', freq: 660, attack: 0.003, decay: 0.14, peak: 0.34 }); + v.tone({ freq: 1320, attack: 0.003, decay: 0.1, peak: 0.12 }); + break; + case 'open': + v.tone({ freq: 520, to: 780, glide: 0.09, attack: 0.005, decay: 0.15, peak: 0.3 }); + v.noise({ filter: 'bandpass', freq: 1200, q: 1.5, attack: 0.004, decay: 0.06, peak: 0.6 }); + break; + case 'close': + v.tone({ freq: 700, to: 470, glide: 0.09, attack: 0.005, decay: 0.13, peak: 0.28 }); + break; + case 'menu': + v.tone({ freq: 196, to: 165, glide: 0.12, attack: 0.006, decay: 0.16, peak: 0.42 }); + // An octave up, so laptop speakers hear the low tone too. + v.tone({ freq: 392, to: 330, glide: 0.12, attack: 0.006, decay: 0.14, peak: 0.12 }); + v.noise({ filter: 'bandpass', freq: 3000, q: 2, attack: 0.001, decay: 0.01, peak: 0.8 }); + break; + case 'confirm': + v.tone({ type: 'triangle', freq: E5, attack: 0.003, decay: 0.16, peak: 0.26 }); + v.tone({ type: 'triangle', freq: A5, at: 0.07, attack: 0.003, decay: 0.22, peak: 0.26 }); + break; + case 'error': { + const lp = s.ctx.createBiquadFilter(); + lp.type = 'lowpass'; + lp.frequency.value = 1500; + lp.connect(v.out); + v.tone({ type: 'triangle', freq: A4, attack: 0.004, decay: 0.16, peak: 0.34, via: lp }); + v.tone({ type: 'triangle', freq: F4, at: 0.11, attack: 0.004, decay: 0.24, peak: 0.34, via: lp }); + break; + } + case 'toggle': + v.tone({ freq: 880, attack: 0.002, decay: 0.04, peak: 0.2 }); + break; + } + return v.done(); +} + +/** A camera fly's whoosh: band-passed pink noise rising then falling, panned across. */ +export function whooshVoice(s: Synth, dest: AudioNode, t: number, seconds: number, pan: number): Voice { + const { ctx, buffers, rng } = s; + const v = new VoiceBuilder(s, dest, t, 'ui'); + const d = Math.max(0.25, Math.min(2.5, seconds)); + const src = ctx.createBufferSource(); + src.buffer = buffers.pink; + const bp = ctx.createBiquadFilter(); + bp.type = 'bandpass'; + bp.Q.value = 1.4; + bp.frequency.setValueAtTime(300, t); + bp.frequency.exponentialRampToValueAtTime(1600, t + d * 0.4); + bp.frequency.exponentialRampToValueAtTime(500, t + d); + const g = ctx.createGain(); + g.gain.setValueAtTime(0, t); + g.gain.linearRampToValueAtTime(0.7, t + d * 0.4); + g.gain.linearRampToValueAtTime(0, t + d); + const p = createPan(ctx, -pan); + if (p.pan) p.pan.linearRampToValueAtTime(Math.max(-1, Math.min(1, pan)), t + d); + src.connect(bp).connect(g).connect(p.node).connect(v.out); + src.start(t, between(rng, 0, Math.max(0, buffers.pink.duration - d - 0.1))); + v.add(src, t, t + d + 0.02, true); + return v.done(); +} + +/** The four event cues. */ +export function cueVoice(s: Synth, dest: AudioNode, t: number, cue: EventCue, pan = 0): Voice { + const { ctx } = s; + const v = new VoiceBuilder(s, dest, t, 'event'); + const p = createPan(ctx, pan); + p.node.connect(v.out); + const via = p.node; + switch (cue) { + case 'agentLive': + v.tone({ freq: C5, attack: 0.008, decay: 0.9, peak: 0.22, via }); + v.tone({ freq: G5, at: 0.04, attack: 0.008, decay: 0.9, peak: 0.17, via }); + v.tone({ freq: C6, at: 0.04, attack: 0.01, decay: 0.5, peak: 0.04, via }); + break; + case 'needsApproval': + for (const at of [0, 0.22]) { + // An FM bell: carrier 880 Hz, modulator 1210 Hz, index 2 decaying. + const tt = t + at; + const car = v.tone({ freq: 880, at, attack: 0.002, decay: 0.7, peak: 0.24, via }); + const mod = ctx.createOscillator(); + mod.frequency.value = 1210; + const depth = ctx.createGain(); + depth.gain.setValueAtTime(2 * 1210, tt); + depth.gain.setTargetAtTime(0, tt, 0.08); + mod.connect(depth).connect(car.frequency); + v.add(mod, tt, tt + 0.6); + } + break; + case 'incident': + v.tone({ freq: 196, to: 165, glide: 0.12, attack: 0.004, decay: 0.22, peak: 0.4, via }); + v.tone({ freq: 392, to: 330, glide: 0.12, attack: 0.004, decay: 0.18, peak: 0.14, via }); + v.noise({ buffer: s.buffers.brown, filter: 'lowpass', freq: 120, attack: 0.004, decay: 0.15, peak: 1.2 }); + v.tone({ type: 'triangle', freq: G4, at: 0.26, attack: 0.01, decay: 0.4, peak: 0.17, via }); + v.tone({ type: 'triangle', freq: E4, at: 0.46, attack: 0.01, decay: 0.6, peak: 0.17, via }); + break; + case 'taskDone': + [C5, E5, G5, C6].forEach((freq, i) => v.tone({ type: 'triangle', freq, at: i * 0.07, attack: 0.004, decay: i === 3 ? 0.7 : 0.3, peak: 0.2 - i * 0.012, via })); + break; + } + return v.done(); +} + +interface Species { + curve: [number, number, number]; + chirps: [number, number]; + dur: [number, number]; + gap: [number, number]; +} + +/** Three made-up songbirds: a rising warbler, a falling sparrow, a low finch. */ +export const SPECIES: readonly Species[] = [ + { curve: [3200, 4600, 3800], chirps: [3, 6], dur: [0.06, 0.09], gap: [0.04, 0.08] }, + { curve: [4300, 3500, 4000], chirps: [2, 4], dur: [0.08, 0.12], gap: [0.08, 0.14] }, + { curve: [2700, 3500, 2900], chirps: [2, 3], dur: [0.1, 0.12], gap: [0.1, 0.18] }, +]; + +/** One bird call: 2–6 chirps on one oscillator (a pitch curve and a short envelope each). */ +export function birdVoice(s: Synth, dest: AudioNode, t: number, species: number, pan: number, peak: number): Voice { + const { ctx, rng } = s; + const sp = SPECIES[Math.abs(species) % SPECIES.length]!; + const v = new VoiceBuilder(s, dest, t, 'bird'); + const p = createPan(ctx, pan); + p.node.connect(v.out); + const osc = ctx.createOscillator(); + const g = ctx.createGain(); + g.gain.value = 0; + osc.connect(g).connect(p.node); + const jitter = between(rng, 0.92, 1.08); + const n = Math.round(between(rng, sp.chirps[0], sp.chirps[1])); + let at = t; + for (let i = 0; i < n; i++) { + const d = between(rng, sp.dur[0], sp.dur[1]); + const curve = new Float32Array(sp.curve.map((f) => f * jitter * between(rng, 0.97, 1.03))); + osc.frequency.setValueCurveAtTime(curve, at, d); + g.gain.setValueAtTime(0, at); + g.gain.linearRampToValueAtTime(peak * between(rng, 0.6, 1), at + 0.005); + g.gain.setTargetAtTime(0, at + 0.005, 0.08 / 4); + at += d + between(rng, sp.gap[0], sp.gap[1]); + } + v.add(osc, t, at + 0.05); + return v.done(); +} diff --git a/apps/studio/src/audio/testing.ts b/apps/studio/src/audio/testing.ts new file mode 100644 index 0000000..04fd981 --- /dev/null +++ b/apps/studio/src/audio/testing.ts @@ -0,0 +1,215 @@ +/** + * Test helpers: a minimal fake of the Web Audio nodes the engine uses (it + * records automation; it makes no sound) and a fake page environment with a + * manual clock and timers. Tests only. Real rendering is checked in a real + * browser by `check/render-check.mjs` (OfflineAudioContext). + */ +import type { EngineEnv } from './engine'; + +export interface ParamEvent { + type: 'set' | 'linear' | 'exp' | 'target' | 'curve' | 'cancel'; + v: number; + t: number; +} + +export class FakeParam { + events: ParamEvent[] = []; + constructor(public value = 0) {} + setValueAtTime(v: number, t: number) { + this.events.push({ type: 'set', v, t }); + this.value = v; + return this; + } + linearRampToValueAtTime(v: number, t: number) { + this.events.push({ type: 'linear', v, t }); + this.value = v; + return this; + } + exponentialRampToValueAtTime(v: number, t: number) { + this.events.push({ type: 'exp', v, t }); + this.value = v; + return this; + } + setTargetAtTime(v: number, t: number) { + this.events.push({ type: 'target', v, t }); + this.value = v; + return this; + } + setValueCurveAtTime(c: ArrayLike, t: number) { + this.events.push({ type: 'curve', v: c[c.length - 1] ?? 0, t }); + return this; + } + cancelScheduledValues(t: number) { + this.events.push({ type: 'cancel', v: this.value, t }); + return this; + } +} + +class FakeNode { + readonly outputs: unknown[] = []; + constructor(readonly ctx: FakeAudioContext, readonly kind: string) {} + connect(dest: T): T { + this.outputs.push(dest); + return dest; + } + disconnect() { + this.outputs.length = 0; + } +} + +class FakeSource extends FakeNode { + started: number | null = null; + stopped: number | null = null; + onended: (() => void) | null = null; + start(t = 0) { + if (this.started !== null) throw new Error('InvalidStateError: start called twice'); + this.started = t; + } + stop(t = 0) { + if (this.started === null) throw new Error('InvalidStateError: stop before start'); + this.stopped = t; + } +} + +class FakeBuffer { + private readonly data: Float32Array[]; + constructor( + readonly numberOfChannels: number, + readonly length: number, + readonly sampleRate: number, + ) { + this.data = Array.from({ length: numberOfChannels }, () => new Float32Array(length)); + } + get duration() { + return this.length / this.sampleRate; + } + getChannelData(i: number) { + return this.data[i]!; + } +} + +export class FakeAudioContext { + currentTime = 0; + readonly sampleRate = 8000; + state: 'running' | 'suspended' | 'closed' = 'running'; + onstatechange: (() => void) | null = null; + readonly destination = new FakeNode(this, 'destination'); + /** Every node created, in order. */ + readonly nodes: FakeNode[] = []; + readonly params: FakeParam[] = []; + + private param(v: number) { + const p = new FakeParam(v); + this.params.push(p); + return p; + } + private add(n: T): T { + this.nodes.push(n); + return n; + } + createGain() { + return this.add(Object.assign(new FakeNode(this, 'gain'), { gain: this.param(1) })); + } + createOscillator() { + return this.add(Object.assign(new FakeSource(this, 'oscillator'), { type: 'sine', frequency: this.param(440), detune: this.param(0) })); + } + createBufferSource() { + return this.add(Object.assign(new FakeSource(this, 'buffer'), { buffer: null as FakeBuffer | null, loop: false, playbackRate: this.param(1) })); + } + createBiquadFilter() { + return this.add(Object.assign(new FakeNode(this, 'biquad'), { type: 'lowpass', frequency: this.param(350), Q: this.param(1), gain: this.param(0) })); + } + createDynamicsCompressor() { + return this.add( + Object.assign(new FakeNode(this, 'compressor'), { threshold: this.param(-24), knee: this.param(30), ratio: this.param(12), attack: this.param(0.003), release: this.param(0.25) }), + ); + } + createConvolver() { + return this.add(Object.assign(new FakeNode(this, 'convolver'), { buffer: null as FakeBuffer | null, normalize: true })); + } + createStereoPanner() { + return this.add(Object.assign(new FakeNode(this, 'panner'), { pan: this.param(0) })); + } + createBuffer(channels: number, length: number, sampleRate: number) { + return new FakeBuffer(channels, length, sampleRate); + } + resume() { + this.setState('running'); + return Promise.resolve(); + } + suspend() { + this.setState('suspended'); + return Promise.resolve(); + } + private setState(s: FakeAudioContext['state']) { + if (this.state === s) return; + this.state = s; + this.onstatechange?.(); + } + /** Sources created and not yet stopped by `t`. */ + sourcesPlayingAt(t: number) { + return this.nodes.filter((n): n is FakeSource => n instanceof FakeSource && n.started !== null && n.started <= t && (n.stopped === null || n.stopped > t)); + } +} + +export interface FakeEnv extends EngineEnv { + contexts: FakeAudioContext[]; + clock: { ms: number }; + /** Advance wall time and run due timers. */ + advance(ms: number): void; + store: Map; +} + +export function fakeEnv(opts: { supported?: boolean; stored?: unknown } = {}): FakeEnv { + const clock = { ms: 1000 }; + const timers = new Map void }>(); + let nextId = 1; + const store = new Map(); + if (opts.stored !== undefined) store.set('runloop.audio.v1', JSON.stringify(opts.stored)); + const contexts: FakeAudioContext[] = []; + const env: FakeEnv = { + supported: opts.supported ?? true, + contexts, + clock, + store, + createContext: () => { + const c = new FakeAudioContext(); + contexts.push(c); + return c as unknown as AudioContext; + }, + storage: { getItem: (k) => store.get(k) ?? null, setItem: (k, v) => void store.set(k, v) }, + now: () => clock.ms, + setInterval: (fn, ms) => { + const id = nextId++; + timers.set(id, { at: clock.ms + ms, every: ms, fn }); + return id as unknown as ReturnType; + }, + clearInterval: (id) => void timers.delete(id as unknown as number), + setTimeout: (fn, ms) => { + const id = nextId++; + timers.set(id, { at: clock.ms + ms, every: null, fn }); + return id as unknown as ReturnType; + }, + clearTimeout: (id) => void timers.delete(id as unknown as number), + advance: (ms) => { + const end = clock.ms + ms; + // The audio clock moves with the wall clock while its context runs. + const moveTo = (to: number) => { + for (const c of contexts) if (c.state === 'running') c.currentTime += (to - clock.ms) / 1000; + clock.ms = to; + }; + for (;;) { + let next: [number, { at: number; every: number | null; fn: () => void }] | null = null; + for (const e of timers) if (e[1].at <= end && (!next || e[1].at < next[1].at)) next = e; + if (!next) break; + const [id, t] = next; + moveTo(t.at); + if (t.every) t.at += t.every; + else timers.delete(id); + t.fn(); + } + moveTo(end); + }, + }; + return env; +} diff --git a/apps/studio/src/audio/types.ts b/apps/studio/src/audio/types.ts new file mode 100644 index 0000000..04c812b --- /dev/null +++ b/apps/studio/src/audio/types.ts @@ -0,0 +1,74 @@ +/** + * The soundscape's public types (contract §4b). Plain types only: the town + * renderer reports a `TownListener` of the same shape and never imports audio. + */ +import type { AgentAttention, ConnectionHealth } from '@runloop-world/events'; + +export type UiSound = 'hover' | 'select' | 'open' | 'close' | 'menu' | 'confirm' | 'error' | 'toggle'; +export const UI_SOUNDS: readonly UiSound[] = ['hover', 'select', 'open', 'close', 'menu', 'confirm', 'error', 'toggle']; + +/** Sounds for real agent events only (never for simulated activity). */ +export type EventCue = 'agentLive' | 'incident' | 'taskDone' | 'needsApproval'; +export const EVENT_CUES: readonly EventCue[] = ['agentLive', 'incident', 'taskDone', 'needsApproval']; + +/** Per-viewer sound settings (localStorage `runloop.audio.v1`). Volumes are 0…1. */ +export interface AudioSettings { + /** Sound on (the HUD speaker toggle; off = mute). */ + enabled: boolean; + master: number; + ambient: number; + ui: number; + events: number; + /** Reduce sound: no ambient bed, no hover ticks, UI −6 dB, only the needs-approval, incident and task-done cues. */ + reduce: boolean; +} + +export type AudioSourceKind = 'murmur' | 'water' | 'green' | 'road' | 'field'; + +/** What the soundscape hears, about four times a second (the town's `onListener`). */ +export interface AudioListener { + /** Camera target in town cells. */ + x: number; + z: number; + /** 0 = the whole town … 1 = street level and closer. */ + zoom01: number; + /** Degrees. */ + az: number; + entered: string | null; + level: 'town' | 'district' | 'street' | 'interior'; + /** Minute of day (fractional). */ + minute: number; + /** 0 = day … 1 = night. */ + night: number; + /** The world clock is paused. */ + paused: boolean; + /** Nearby sources with a gain (nearness, 0…1) and a pan (−1 left … 1 right). Murmur = simulated occupancy only. */ + sources: { kind: AudioSourceKind; id: string; gain: number; pan: number }[]; +} + +export type AudioState = 'locked' | 'running' | 'suspended' | 'unsupported'; + +export interface RunloopAudio { + /** locked = no AudioContext yet (before the first gesture); unsupported = SSR or no Web Audio. */ + readonly state: AudioState; + /** From a user gesture; idempotent. Creates the AudioContext only when sound is enabled. */ + unlock(): void; + ui(sound: UiSound): void; + /** A camera move's whoosh (not for cuts; never under reduced motion). `pan` −1…1 sweeps from −pan to +pan. */ + whoosh(durationMs: number, pan?: number): void; + /** A real agent event's cue. false = not played (limited, disabled, locked or hidden). */ + cue(cue: EventCue, opts?: { agentId?: string; pan?: number }): boolean; + setListener(l: AudioListener): void; + setReducedMotion(v: boolean): void; + getSettings(): AudioSettings; + update(p: Partial): void; + /** Called when settings or state change. Returns the unsubscribe. */ + subscribe(fn: () => void): () => void; +} + +/** The tracker's view of one agent, before an event changes it. */ +export interface AgentAudioState { + health: ConnectionHealth; + attention: AgentAttention; + openIncidents: number; +} diff --git a/apps/studio/src/audio/world.test.ts b/apps/studio/src/audio/world.test.ts new file mode 100644 index 0000000..4c9cc70 --- /dev/null +++ b/apps/studio/src/audio/world.test.ts @@ -0,0 +1,132 @@ +import type { WorldEvent } from '@runloop-world/events'; +import { describe, expect, it } from 'vitest'; +import { createWorldStore } from '../world/store'; +import { DOCS_DESK, ENGINEERING_HOUSE, commonsTown, fixtureEvents, fixtureSnapshot } from '../world/test-utils'; +import { AudioEngine } from './engine'; +import { fakeEnv } from './testing'; +import type { EventCue } from './types'; +import { RESYNC_QUIET_MS, attachWorldAudio, type WorldCueSink } from './world'; + +const NOW = 1_700_000_000_000; + +function recordingSink() { + const calls = { cues: [] as { cue: EventCue; agentId: string | undefined }[], suppress: 0, reduced: [] as boolean[], excluded: [] as string[][] }; + const sink: WorldCueSink = { + cue: (cue, opts) => { + calls.cues.push({ cue, agentId: opts?.agentId }); + return true; + }, + setReducedMotion: (v) => void calls.reduced.push(v), + suppressCues: () => void calls.suppress++, + setExcluded: (ids) => void calls.excluded.push([...ids]), + }; + return { sink, calls }; +} + +function liveStore() { + const store = createWorldStore(); + store.getState().setTown(commonsTown()); + store.getState().applySnapshot(fixtureSnapshot(), NOW); + store.getState().setStatus('live'); + return store; +} + +const tail = () => { + const snap = fixtureSnapshot(); + return fixtureEvents().filter((e) => e.seq > snap.seq); +}; + +let extraSeq = 0; +function connection(store: ReturnType, agentId: string, health: string, attention = 'none'): WorldEvent { + const s = store.getState().snapshot!; + return { + seq: s.seq + 1, + tick: s.tick + 1, + type: 'agent.connection_changed', + prov: 'system', + cmdId: `conn:${agentId}:x${++extraSeq}`, + actor: agentId, + p: { agentId, health, attention }, + } as WorldEvent; +} + +describe('attachWorldAudio', () => { + it('a recorded day: ~100 simulated events make no sound; the one real permission request cues needsApproval once', () => { + const store = liveStore(); + const { sink, calls } = recordingSink(); + const off = attachWorldAudio(store, sink); + const events = tail(); + expect(events.filter((e) => e.prov === 'simulation').length).toBeGreaterThan(90); + store.getState().applyEvents(events, NOW + 50); + expect(calls.cues).toEqual([{ cue: 'needsApproval', agentId: ENGINEERING_HOUSE }]); + off(); + }); + + it('never cues what was already in the snapshot or log when it attached', () => { + const store = liveStore(); + store.getState().applyEvents(tail(), NOW + 50); + const { sink, calls } = recordingSink(); + attachWorldAudio(store, sink); + expect(calls.cues).toEqual([]); + expect(calls.suppress).toBe(1); + }); + + it('knows each agent from the snapshot: an already-live agent does not "go live" again; a waiting one does', () => { + const store = liveStore(); + const { sink, calls } = recordingSink(); + attachWorldAudio(store, sink); + store.getState().applyEvents([connection(store, ENGINEERING_HOUSE, 'live')], NOW); + expect(calls.cues).toEqual([]); + store.getState().applyEvents([connection(store, DOCS_DESK, 'live')], NOW); + expect(calls.cues).toEqual([{ cue: 'agentLive', agentId: DOCS_DESK }]); + }); + + it('only while the world is live: a reconnect backfill is silent, and returning to live suppresses for a moment', () => { + const store = liveStore(); + const { sink, calls } = recordingSink(); + attachWorldAudio(store, sink); + expect(calls.suppress).toBe(1); + store.getState().setStatus('reconnecting'); + store.getState().applyEvents([connection(store, DOCS_DESK, 'live')], NOW); + expect(calls.cues).toEqual([]); + store.getState().setStatus('live'); + expect(calls.suppress).toBe(2); + store.getState().setStatus('disabled'); + store.getState().applyEvents([connection(store, DOCS_DESK, 'live', 'permission_requested')], NOW); + expect(calls.cues).toEqual([]); + }); + + it('a resync snapshot re-seeds the agents and suppresses', () => { + const store = liveStore(); + const { sink, calls } = recordingSink(); + attachWorldAudio(store, sink); + const fresh = fixtureSnapshot(); + fresh.agents[1]!.health = 'live'; + store.getState().applySnapshot(fresh, NOW + 10); + expect(calls.suppress).toBe(2); + store.getState().applyEvents([connection(store, DOCS_DESK, 'live')], NOW); + expect(calls.cues).toEqual([]); // already live in the new snapshot + }); + + it('passes reduced motion and the agent places (never a murmur source) to the engine', () => { + const store = liveStore(); + const { sink, calls } = recordingSink(); + attachWorldAudio(store, sink); + expect(calls.excluded.at(-1)).toEqual(expect.arrayContaining(['bld_agent_hq_1', 'lot_agent_1', 'bld_agent_hq_4', 'lot_agent_4'])); + store.getState().setReducedMotion(true); + expect(calls.reduced).toEqual([false, true]); + }); + + it('with the real engine: suppressed for 3 s after attaching, then played and paced', () => { + const env = fakeEnv(); + const audio = new AudioEngine(env); + audio.unlock(); + const store = liveStore(); + attachWorldAudio(store, audio); + store.getState().applyEvents([connection(store, DOCS_DESK, 'live')], NOW); + expect(audio.debug().lastCues.at(-1)).toMatchObject({ cue: 'agentLive', result: 'suppressed' }); + env.advance(RESYNC_QUIET_MS + 10); + store.getState().applyEvents([connection(store, DOCS_DESK, 'live', 'permission_requested')], NOW); + expect(audio.debug().lastCues.at(-1)).toMatchObject({ cue: 'needsApproval', agentId: DOCS_DESK, result: 'play' }); + }); +}); diff --git a/apps/studio/src/audio/world.ts b/apps/studio/src/audio/world.ts new file mode 100644 index 0000000..3581e50 --- /dev/null +++ b/apps/studio/src/audio/world.ts @@ -0,0 +1,89 @@ +/** + * The world's event cues (contract §4b): follows the world store's log and + * plays a cue for a live, real agent event (`classifyCue`). + * + * - It keeps its own agent state (health, attention, open incidents), seeded + * from the snapshot and folded from each event in seq order, so it knows + * what each event changed. + * - Nothing from before it attached, from a snapshot (connect or resync), or + * from a reconnect's backfill makes a sound: cues need the world `live` and + * are suppressed for 3 s after each snapshot and each return to `live`. + * - A disabled world (RUNLOOP_WORLD=off) never cues. + * - It also passes the store's reduced-motion flag (no whoosh) and the agent + * buildings and lots (never a murmur source) to the engine. + */ +import type { WorldSnapshot } from '@runloop-world/events'; +import type { WorldStore, WorldUiState } from '../world/store'; +import { classifyCue, cueAgentId, nextAgentState } from './cues'; +import { getAudio, getAudioEngine } from './engine'; +import type { AgentAudioState, RunloopAudio } from './types'; + +/** After a snapshot or a reconnect, events are catching up, not news. */ +export const RESYNC_QUIET_MS = 3000; + +/** What the world needs from the engine (the engine, or a fake in tests). */ +export interface WorldCueSink { + cue: RunloopAudio['cue']; + setReducedMotion: RunloopAudio['setReducedMotion']; + suppressCues?(ms: number): void; + setExcluded?(ids: Iterable): void; +} + +export function attachWorldAudio(store: WorldStore, sink: WorldCueSink = getAudioEngine() ?? getAudio()): () => void { + const agents = new Map(); + let snap: WorldSnapshot | null = null; + let lastSeq = 0; + let status: WorldUiState['status'] | null = null; + let reduced: boolean | null = null; + + const excludePlaces = (s: WorldSnapshot) => { + const ids: string[] = []; + for (const a of s.agents) ids.push(a.buildingId, a.lotId); + sink.setExcluded?.(ids); + }; + + const seed = (s: WorldSnapshot) => { + agents.clear(); + for (const a of s.agents) agents.set(a.agentId, { health: a.health, attention: a.attention, openIncidents: a.openIncidents }); + lastSeq = s.seq; + sink.suppressCues?.(RESYNC_QUIET_MS); + excludePlaces(s); + }; + + const onState = (s: WorldUiState) => { + if (s.reducedMotion !== reduced) { + reduced = s.reducedMotion; + sink.setReducedMotion(reduced); + } + if (s.status !== status) { + if (s.status === 'live' && status !== null) sink.suppressCues?.(RESYNC_QUIET_MS); + status = s.status; + } + if (s.snapshot !== snap) { + snap = s.snapshot; + if (snap) seed(snap); + return; + } + if (!snap) return; + const log = s.log; + let i = log.length; + while (i > 0 && log[i - 1]!.seq > lastSeq) i--; + let housed = false; + for (; i < log.length; i++) { + const e = log[i]!; + lastSeq = e.seq; + const id = cueAgentId(e); + if (!id) continue; + const before = agents.get(id); + const cue = classifyCue(e, before); + const after = nextAgentState(before, e); + if (after) agents.set(id, after); + if (e.type === 'agent.housed') housed = true; + if (cue && s.status === 'live') sink.cue(cue, { agentId: id }); + } + if (housed) excludePlaces(snap); + }; + + onState(store.getState()); + return store.subscribe(onState); +} diff --git a/apps/studio/src/components/AppShell.module.css b/apps/studio/src/components/AppShell.module.css index 2a9db09..46efcf1 100644 --- a/apps/studio/src/components/AppShell.module.css +++ b/apps/studio/src/components/AppShell.module.css @@ -9,11 +9,29 @@ .shell[data-rail] { --sidebar-w: 64px; } +/* Other pages use the ModeRail (plan §5); run pages keep the run sidebar. */ +.shell[data-layout='page'] { + --sidebar-w: 76px; +} .shell[data-mobile] { grid-template-columns: minmax(0, 1fr); height: auto; min-height: 100dvh; } +/* The town is the whole screen: no sidebar, no top bar (immersive contract §0). */ +.shell[data-layout='town'] { + display: block; + position: fixed; + inset: 0; + width: 100vw; + height: 100dvh; + overflow: hidden; + background: var(--rl-world-bg); +} +.shell[data-layout='town'] .main { + position: absolute; + inset: 0; +} .skip { position: absolute; @@ -90,6 +108,30 @@ text-align: left; } +.townMain { + position: relative; + flex: 1; + min-height: 0; + outline: none; +} + +.backToTown { + display: inline-flex; + align-items: center; + gap: 6px; + height: 30px; + padding: 0 10px; + border-radius: var(--rl-radius-control); + background: var(--rl-surface); + color: var(--rl-text); + font-size: var(--rl-text-xs); + font-weight: 500; + white-space: nowrap; +} +.backToTown:hover { + background: var(--rl-surface-2); +} + .content { flex: 1; min-height: 0; @@ -104,3 +146,7 @@ max-height: none; border-radius: 0 var(--rl-radius-card) var(--rl-radius-card) 0; } +/* The town is one big region: focusing it (skip link, a click) needs no ring around the whole screen. */ +.townMain:focus-visible { + box-shadow: none; +} diff --git a/apps/studio/src/components/AppShell.tsx b/apps/studio/src/components/AppShell.tsx index d3fd1d7..962724e 100644 --- a/apps/studio/src/components/AppShell.tsx +++ b/apps/studio/src/components/AppShell.tsx @@ -1,4 +1,5 @@ 'use client'; +import Link from 'next/link'; import { usePathname } from 'next/navigation'; import { useEffect, useRef, useState, type ReactNode } from 'react'; import { connectRun } from '../state/connect'; @@ -12,47 +13,77 @@ import { Button } from '../ui/Button'; import { CommandPalette } from '../ui/CommandPalette'; import { ConnectionStatus } from '../ui/ConnectionStatus'; import { Dialog } from '../ui/Dialog'; +import { Icon } from '../ui/Icon'; import { Kbd, modKey } from '../ui/Kbd'; import { ToastRegion } from '../ui/Toast'; import { MOBILE_QUERY, NARROW_QUERY, useMediaQuery } from '../ui/useMediaQuery'; +import { connectWorld } from '../world/connect'; +import { useWorld, useWorldStore, WorldProvider } from '../world/context'; +import { worldHref } from '../world/link'; +import { agentOfRun } from '../world/selectors'; +import { createWorldStore, type WorldStore } from '../world/store'; +import AddToWorldDialog from './agents/AddToWorldDialog'; import styles from './AppShell.module.css'; import ArtifactViewer from './ArtifactViewer'; import StudioSidebar from './StudioSidebar'; import { useStudioCommands } from './useStudioCommands'; +import ModeRail from './world/ModeRail'; +import { ShellActionsContext } from './shellActions'; +import { useWorldCommands } from './world/useWorldCommands'; /** - * Root client component: one studio store for the whole app (so a live run - * keeps streaming across pages), the runtime connection, and the chrome. + * Root client component: one studio store (sessions and replay; a live run + * keeps streaming across pages), one world store (the town, connected for + * the whole app), the runtime connections, and the chrome. + * + * Three layouts: the town at `/` (the world edge to edge with its game HUD: + * no sidebar, no top bar), run pages (/world, /traces: the run sidebar, + * unchanged), and other pages (ModeRail + the page under a top bar). */ export default function AppShell({ children }: { children: ReactNode }) { const [store] = useState(() => createStudioStore()); + const [world] = useState(() => createWorldStore()); return ( - {children} + + + {children} + + ); } -function Shell({ store, children }: { store: StudioStore; children: ReactNode }) { +type Layout = 'town' | 'run' | 'page'; + +function Shell({ store, world, children }: { store: StudioStore; world: WorldStore; children: ReactNode }) { const pathname = usePathname() ?? '/'; + const path = pathname.replace(/\/+$/, '') || '/'; + const layout: Layout = path === '/' ? 'town' : isRunPage(pathname) ? 'run' : 'page'; const runId = useStudio((s) => s.runId); const connectNonce = useStudio((s) => s.connectNonce); const apiKey = useStudio((s) => s.apiBaseOverride) ?? ''; + const reducedMotion = useStudio((s) => s.reducedMotion); const retry = useReconnectNonce(store); const mobile = useMediaQuery(MOBILE_QUERY); const narrow = useMediaQuery(NARROW_QUERY); const [paletteOpen, setPaletteOpen] = useState(false); const [drawerOpen, setDrawerOpen] = useState(false); + const traceSheet = useWorld((s) => s.overlay === 'trace'); // Order matters: preferences (runtime URL) before the deep link and polling. usePrefs(store); useDeepLinkOnce(store); useRuntimePolling(store, apiKey); usePlaybackDriver(store); - useKeyboardShortcuts(store); + // The town has its own keys; the trace sheet over it uses the run shortcuts (play, step, live). + useKeyboardShortcuts(store, layout !== 'town' || traceSheet); useUrlSync(store, pathname); useOnboardingTracker(store, pathname); useEffect(() => (runId ? connectRun(store, runId) : undefined), [store, runId, retry, connectNonce]); + // The town stays connected on every page (the ticker, ⌘K and Back to town are instant). + useEffect(() => connectWorld(world), [world, apiKey]); + useEffect(() => world.getState().setReducedMotion(reducedMotion), [world, reducedMotion]); useEffect(() => setDrawerOpen(false), [pathname]); useEffect(() => { @@ -75,53 +106,90 @@ function Shell({ store, children }: { store: StudioStore; children: ReactNode }) return () => window.removeEventListener('keydown', onKey); }, []); - const commands = useStudioCommands(pathname); + const commands = [...useWorldCommands(pathname), ...useStudioCommands(pathname)]; + const [actions] = useState(() => ({ openPalette: () => setPaletteOpen(true), openMenu: () => setDrawerOpen(true) })); return ( -
+
Skip to content - {!mobile && ( + {!mobile && layout !== 'town' && ( )}
- setPaletteOpen(true)} onOpenMenu={() => setDrawerOpen(true)} /> -
- {children} -
+ {layout === 'town' ? ( +
+ {children} +
+ ) : ( + <> + +
+ {children} +
+ + )}
- {mobile && ( + {mobile && layout !== 'town' && ( setDrawerOpen(false)} labelledBy="drawer-title" placement="left" className={styles.drawer}>

Navigation

- setDrawerOpen(false)} /> + {layout === 'run' ? ( + setDrawerOpen(false)} /> + ) : ( + setDrawerOpen(false)} /> + )} )} setPaletteOpen(false)} commands={commands} /> +
); } +/** + * Add to World as a dialog (with its lot list): from another page, or on the + * town without the 3D view (no WebGL, the 2D map). In 3D it is lot picking. + */ +function WorldDialogs() { + const world = useWorldStore(); + const add = useWorld((s) => s.addToWorld); + return world.getState().closeAddToWorld()} />; +} + function TopBar({ pathname, mobile, onOpenPalette, onOpenMenu }: { pathname: string; mobile: boolean; onOpenPalette: () => void; onOpenMenu: () => void }) { const area = areaOf(pathname); const runs = useStudio((s) => s.runs); const runId = useStudio((s) => s.runId); const run = useStudio((s) => s.run); const conn = connectionView(useStudio((s) => connectionState(s))); + const agents = useWorld((s) => s.agents); const [mod, setMod] = useState('Ctrl'); useEffect(() => setMod(modKey()), []); - const crumbs: Crumb[] = [{ label: AREA_LABEL[area] ?? 'Studio', href: area }]; - // Phones show the area only; the run title is in the page header. - if (!mobile && isRunPage(pathname) && runId) { - const n = runNumber(runs, runId); - crumbs.push({ label: `${n !== null ? `Run #${n}` : 'Run'}${run ? ` · ${run.title}` : ''}` }); + const runPage = isRunPage(pathname); + const owner = runPage ? agentOfRun(agents, runId) : null; + const crumbs: Crumb[] = []; + if (owner && !mobile) { + // A run of a real agent: it happened in that agent's building (plan §5). + crumbs.push( + { label: 'Agent District', href: worldHref({ mode: 'agents' }) }, + { label: owner.agent.displayName, href: worldHref({ mode: 'agents', focus: { kind: 'agent', id: owner.agent.agentId } }) }, + { label: `Session #${owner.session}${run ? ` · ${run.title}` : ''}` }, + ); + } else { + crumbs.push({ label: AREA_LABEL[area] ?? 'Studio', href: area }); + // Phones show the area only; the run title is in the page header. + if (!mobile && runPage && runId) { + const n = runNumber(runs, runId); + crumbs.push({ label: `${n !== null ? `Run #${n}` : 'Run'}${run ? ` · ${run.title}` : ''}` }); + } } return ( @@ -129,6 +197,13 @@ function TopBar({ pathname, mobile, onOpenPalette, onOpenMenu }: { pathname: str {mobile &&
diff --git a/apps/studio/src/components/RunHeader.module.css b/apps/studio/src/components/RunHeader.module.css index 3d81973..feff903 100644 --- a/apps/studio/src/components/RunHeader.module.css +++ b/apps/studio/src/components/RunHeader.module.css @@ -123,3 +123,15 @@ flex-wrap: wrap; } } + +/* The in-world trace sheet's compact header. */ +.header[data-variant='sheet'] { + gap: var(--rl-space-2) var(--rl-space-3); + padding: 0; +} +.header[data-variant='sheet'] .title { + flex-basis: 240px; +} +.header[data-variant='sheet'] .h1 { + font-size: var(--rl-text-lg); +} diff --git a/apps/studio/src/components/RunHeader.tsx b/apps/studio/src/components/RunHeader.tsx index 2a33e96..efd2824 100644 --- a/apps/studio/src/components/RunHeader.tsx +++ b/apps/studio/src/components/RunHeader.tsx @@ -16,10 +16,16 @@ import { GlyphIcon, Icon } from '../ui/Icon'; import { Popover } from '../ui/Popover'; import { toast } from '../ui/Toast'; import { CompareWithBaseline, PinButton } from './Baseline'; +import OpenInWorld from './overlays/OpenInWorld'; import styles from './RunHeader.module.css'; -/** Run title, status, provenance and the run's actions; `page` decides the cross-link. */ -export default function RunHeader({ page }: { page: 'world' | 'traces' }) { +/** + * Run title, status, provenance and the run's actions; `page` decides the + * cross-link. `variant="sheet"` is the compact header of the in-world trace + * sheet: an h2 (the town owns the page's h1), no cross-link (the sheet has + * its own World tab and "Full page" link). + */ +export default function RunHeader({ page, variant = 'page' }: { page: 'world' | 'traces'; variant?: 'page' | 'sheet' }) { const store = useStudioStore(); const run = useStudio((s) => s.run); const runs = useStudio((s) => s.runs); @@ -55,14 +61,17 @@ export default function RunHeader({ page }: { page: 'world' | 'traces' }) { } }; + const sheet = variant === 'sheet'; + const Heading = sheet ? 'h2' : 'h1'; + return ( -
+
{n !== null ? `#${n}` : '#'}
-

+ {run?.title ?? 'Loading run…'} -

+ {(subtitle || activity) && (

{subtitle} @@ -122,7 +131,7 @@ export default function RunHeader({ page }: { page: 'world' | 'traces' }) { )} - {page === 'world' ? ( + {sheet ? null : page === 'world' ? ( Investigate in Traces @@ -131,6 +140,7 @@ export default function RunHeader({ page }: { page: 'world' | 'traces' }) { Open Live World )} + {!sheet && } {canStop && ( - + Connect Claude Code - + } > diff --git a/apps/studio/src/components/StudioSidebar.tsx b/apps/studio/src/components/StudioSidebar.tsx index d8078ac..7ddb393 100644 --- a/apps/studio/src/components/StudioSidebar.tsx +++ b/apps/studio/src/components/StudioSidebar.tsx @@ -14,8 +14,9 @@ import { GlyphIcon, Icon } from '../ui/Icon'; import { Sidebar, type NavItem } from '../ui/Sidebar'; import styles from './StudioSidebar.module.css'; -/** Areas that work today; Evaluations stays hidden until it does. */ +/** Areas that work today (the town first); Evaluations stays hidden until it does. */ const AREAS = [ + { href: '/', label: 'Town', icon: 'map' }, { href: ROUTES.overview, label: 'Overview', icon: 'overview' }, { href: ROUTES.world, label: 'Live World', icon: 'world' }, { href: ROUTES.traces, label: 'Traces', icon: 'traces' }, diff --git a/apps/studio/src/components/WorldStage.tsx b/apps/studio/src/components/WorldStage.tsx index 6d3bd16..45b7414 100644 --- a/apps/studio/src/components/WorldStage.tsx +++ b/apps/studio/src/components/WorldStage.tsx @@ -160,11 +160,11 @@ export default function WorldStage({ allow3d = true }: { allow3d?: boolean }) {

)} diff --git a/apps/studio/src/components/agents/AddToWorldDialog.tsx b/apps/studio/src/components/agents/AddToWorldDialog.tsx new file mode 100644 index 0000000..2b9fa86 --- /dev/null +++ b/apps/studio/src/components/agents/AddToWorldDialog.tsx @@ -0,0 +1,284 @@ +'use client'; +import type { AgentSummary } from '@runloop-world/events'; +import Link from 'next/link'; +import { useEffect, useId, useMemo, useRef, useState } from 'react'; +import { CC } from '../../lib/claude-code'; +import { ROUTES } from '../../state/deeplink'; +import { createAgent, WorldApiError } from '../../world/api'; +import { refreshAgents } from '../../world/connect'; +import { useWorld, useWorldStore } from '../../world/context'; +import { Button } from '../../ui/Button'; +import { Dialog } from '../../ui/Dialog'; +import { Icon } from '../../ui/Icon'; +import AddedAgentStatus from './AddedAgentStatus'; +import styles from './agents.module.css'; + +type Step = 1 | 2 | 3 | 'done'; +type Source = { kind: 'arrival'; sessionId: string } | { kind: 'path' }; + +const STEPS = ['Agent', 'Lot', 'Workspace'] as const; + +/** Lot names from their position: "North row · 1". */ +function lotName(id: string): string { + const n = Number(id.replace(/\D+/g, '')); + return Number.isFinite(n) && n > 0 ? `Lot ${n}` : id; +} + +/** + * Add to World (plan §5, UX §8): Claude Code and a name → a free Agent + * District lot → a workspace (an Arrival or a typed folder). The building + * appears dimmed as "Waiting for first event" until a hook really arrives; + * nothing here pretends the agent is connected. + */ +export default function AddToWorldDialog({ open, onClose, fromSessionId }: { open: boolean; onClose: () => void; fromSessionId?: string | null }) { + const id = useId(); + const store = useWorldStore(); + const town = useWorld((s) => s.town); + useWorld((s) => s.rev); + const agents = useWorld((s) => s.snapshot?.agents); + const arrivals = useWorld((s) => s.arrivals); + const agentsApi = useWorld((s) => s.agentsApi); + const [step, setStep] = useState(1); + const [name, setName] = useState(''); + const [lotId, setLotId] = useState(null); + const [source, setSource] = useState({ kind: 'path' }); + const [path, setPath] = useState(''); + const [busy, setBusy] = useState(false); + const [error, setError] = useState(null); + const [created, setCreated] = useState(null); + const nameRef = useRef(null); + + const taken = useMemo(() => new Map((agents ?? []).map((a) => [a.lotId, a.displayName])), [agents]); + const lots = town?.agentLots ?? []; + const firstFree = lots.find((l) => !taken.has(l.id))?.id ?? null; + + useEffect(() => { + if (!open) return; + setStep(1); + setError(null); + setCreated(null); + setBusy(false); + setName((agents?.length ?? 0) === 0 ? 'Engineering House' : ''); + setLotId(firstFree); + setSource(fromSessionId ? { kind: 'arrival', sessionId: fromSessionId } : arrivals[0] ? { kind: 'arrival', sessionId: arrivals[0].sessionId } : { kind: 'path' }); + setPath(''); + // Reset only when the dialog opens (agents, arrivals and the first free lot are read once per opening). + }, [open]); + + const nameOk = name.trim().length > 0 && name.trim().length <= 60; + const lotOk = lotId !== null && !taken.has(lotId); + const sourceOk = source.kind === 'arrival' ? arrivals.some((a) => a.sessionId === source.sessionId) : path.trim().length > 0; + const apiMissing = agentsApi === 'unavailable'; + + const submit = async () => { + if (!lotId) return; + setBusy(true); + setError(null); + try { + const res = await createAgent({ + displayName: name.trim(), + lotId, + workspace: source.kind === 'arrival' ? { fromSessionId: source.sessionId } : { path: path.trim() }, + }); + setCreated(res.agent); + setStep('done'); + void refreshAgents(store); + store.getState().setMode('agents'); + store.getState().focusEntity({ kind: 'agent', id: res.agent.agentId }); + } catch (e) { + if (e instanceof WorldApiError) { + // Send the viewer back to the step the problem is about (the message never contains a path). + const where = e.code === 'lot_unavailable' || e.code === 'unknown_entity' ? 2 : e.code === 'world_error' ? null : 3; + setError(e.message.endsWith('.') ? e.message : `${e.message}.`); + if (where) setStep(where); + } else setError(e instanceof Error ? `Could not reach the runtime: ${e.message}.` : String(e)); + } finally { + setBusy(false); + } + }; + + const next = () => { + setError(null); + if (step === 1 && nameOk) setStep(2); + else if (step === 2 && lotOk) setStep(3); + else if (step === 3 && sourceOk) void submit(); + }; + + return ( + +
{ + e.preventDefault(); + next(); + }} + > +
+

+ Add to World +

+

Give a real Claude Code agent a building in the Agent District. The town keeps living whether it is connected or not.

+ {step !== 'done' && ( +
    + {STEPS.map((label, i) => ( +
  1. i + 1 ? 'done' : 'todo'} aria-current={step === i + 1 ? 'step' : undefined}> + {(step as number) > i + 1 ? : i + 1} + {label} +
    2. + ))} +
+ )} +
+ +
+ {apiMissing && ( +

+ + This runtime does not serve the agents API yet (GET /api/agents answered 404), so agents cannot be added from here. The town still runs. +

+ )} + + {step === 1 && ( +
+ Which agent + +

Claude Code is the only adapter so far.

+ + setName(e.target.value)} + aria-invalid={!nameOk && name.length > 0} + /> +
+ )} + + {step === 2 && ( +
+ A lot in the Agent District + {lots.length === 0 ? ( +

This town has no agent lots.

+ ) : ( +
+ {lots.map((l) => { + const who = taken.get(l.id); + return ( + + ); + })} +
+ )} +

Lots are reserved for real agents: residents never walk onto them.

+
+ )} + + {step === 3 && ( +
+ Its workspace +

Claude Code sessions started inside this folder will bind to the agent. The full path stays on this machine; the world only shows the folder's name.

+ {arrivals.length > 0 && ( + <> + A session Runloop has already seen (Arrivals) + {arrivals.map((a) => ( + + ))} + + )} + + {source.kind === 'path' && ( + setPath(e.target.value)} + /> + )} +
+ )} + + {step === 'done' && created && ( +
+ {/* Live state, not the create answer: the step follows the agent once its first real event arrives. */} + + Or load it from a Runloop checkout with {CC.pluginDir}. Details and --doctor are on Connect. + + } + /> +
+ )} + + {error && ( +

+ + {error} Nothing was saved. +

+ )} +
+ +
+ {step === 'done' ? ( + + ) : ( + <> + + + + )} +
+
+ + ); +} diff --git a/apps/studio/src/components/agents/AddToWorldFlow.tsx b/apps/studio/src/components/agents/AddToWorldFlow.tsx new file mode 100644 index 0000000..9558597 --- /dev/null +++ b/apps/studio/src/components/agents/AddToWorldFlow.tsx @@ -0,0 +1,261 @@ +'use client'; +/** + * Add to World inside the town (immersive contract §4.3). The viewer has + * picked a free lot in the Agent District; this card, anchored to that lot, + * collects the building's name and the workspace (an Arrival or a folder), + * shows how Claude Code connects, and creates the agent. The building then + * stands as an unpainted model, "Waiting for first event", until a real hook + * arrives: nothing here claims a connection. Errors keep their codes + * (`lot_unavailable` sends the viewer back to picking a lot). + * + * `layout="dialog"` is the 2D route (no WebGL, the 2D map view, other pages): + * the dialog with its lot radio group. + */ +import type { AgentSummary } from '@runloop-world/events'; +import { useEffect, useId, useMemo, useRef, useState } from 'react'; +import { CC } from '../../lib/claude-code'; +import { useStudio } from '../../state/context'; +import { createAgent, WorldApiError } from '../../world/api'; +import { refreshAgents } from '../../world/connect'; +import { useWorld, useWorldStore } from '../../world/context'; +import { Button } from '../../ui/Button'; +import { Chip } from '../../ui/Chip'; +import { CommandSnippet } from '../../ui/CommandSnippet'; +import { Icon } from '../../ui/Icon'; +import { lotLabel } from '../world/hud/lots'; +import AddedAgentStatus from './AddedAgentStatus'; +import AddToWorldDialog from './AddToWorldDialog'; +import styles from './agents.module.css'; + +type Source = { kind: 'arrival'; sessionId: string } | { kind: 'path' }; + + +export default function AddToWorldFlow({ + layout, + lotId, + onLotChange, + onClose, + onDone, + onArrived, + fromSessionId = null, +}: { + layout: 'anchored' | 'dialog'; + lotId: string | null; + /** Back to lot picking (null), with the reason when the runtime refused the lot. */ + onLotChange: (lotId: string | null, reason?: string) => void; + onClose: () => void; + /** After "Done": the new agent's id. */ + onDone?: (agentId: string) => void; + /** The new agent's first real event arrived while the last step was open (hand over to its card). */ + onArrived?: (agentId: string) => void; + fromSessionId?: string | null; +}) { + if (layout === 'dialog') return ; + return ; +} + +function AnchoredFlow({ + lotId, + onLotChange, + onClose, + onDone, + onArrived, + fromSessionId, +}: { + lotId: string | null; + onLotChange: (lotId: string | null, reason?: string) => void; + onClose: () => void; + onDone?: (agentId: string) => void; + onArrived?: ((agentId: string) => void) | undefined; + fromSessionId: string | null; +}) { + const id = useId(); + const store = useWorldStore(); + const town = useWorld((s) => s.town); + useWorld((s) => s.rev); + const agents = useWorld((s) => s.snapshot?.agents); + const arrivals = useWorld((s) => s.arrivals); + const agentsApi = useWorld((s) => s.agentsApi); + const ingest = useStudio((s) => s.runtime.ingest); + const [name, setName] = useState(''); + const [source, setSource] = useState({ kind: 'path' }); + const [path, setPath] = useState(''); + const [busy, setBusy] = useState(false); + const [error, setError] = useState(null); + const [created, setCreated] = useState(null); + const nameRef = useRef(null); + + // Defaults once per opening: the first agent's building is "Engineering House". + useEffect(() => { + setName((agents?.length ?? 0) === 0 ? 'Engineering House' : ''); + setSource(fromSessionId ? { kind: 'arrival', sessionId: fromSessionId } : arrivals[0] ? { kind: 'arrival', sessionId: arrivals[0].sessionId } : { kind: 'path' }); + requestAnimationFrame(() => nameRef.current?.focus()); + }, []); + + const lot = lotId ? town?.lotById.get(lotId) : undefined; + const taken = useMemo(() => new Set((agents ?? []).map((a) => a.lotId)), [agents]); + const nameOk = name.trim().length > 0 && name.trim().length <= 60; + const sourceOk = source.kind === 'arrival' ? arrivals.some((a) => a.sessionId === source.sessionId) : path.trim().length > 0; + const apiMissing = agentsApi === 'unavailable'; + // Only a real agent lot of this town that nobody holds (a deep link can name anything). + const lotOk = lotId !== null && lot?.use === 'agent' && !taken.has(lotId); + + const submit = async () => { + if (!lotId || !nameOk || !sourceOk) return; + setBusy(true); + setError(null); + try { + const res = await createAgent({ + displayName: name.trim(), + lotId, + workspace: source.kind === 'arrival' ? { fromSessionId: source.sessionId } : { path: path.trim() }, + }); + setCreated(res.agent); + void refreshAgents(store); + } catch (e) { + if (e instanceof WorldApiError) { + const msg = e.message.endsWith('.') ? e.message : `${e.message}.`; + // The lot went to someone else: back to picking one (the message never contains a path). + if (e.code === 'lot_unavailable' || e.code === 'unknown_entity') onLotChange(null, `${msg} Nothing was saved. Pick another free lot.`); + else setError(`${msg} Nothing was saved.`); + } else setError(`Could not reach the runtime: ${e instanceof Error ? e.message : String(e)}. Nothing was saved.`); + } finally { + setBusy(false); + } + }; + + if (created) { + return ( +
+ {/* Live state, not the create answer: from the first real event this hands over to the agent's own card. */} + +
+ +
+
+ ); + } + + return ( +
{ + e.preventDefault(); + void submit(); + }} + > +
+

+ + Add to World +

+

+ {lotId ? lotLabel(lotId) : 'No lot'} + {lot ? ` · ${lot.rect[2]}×${lot.rect[3]} cells` : ''} · {lotOk ? 'free' : lot?.use === 'agent' ? 'taken' : 'not an agent lot'} + +

+
+ + {apiMissing && ( +

+ + This runtime does not serve the agents API (GET /api/agents answered 404), so agents cannot be added. The town still runs. +

+ )} + + + setName(e.target.value)} + aria-invalid={!nameOk && name.length > 0} + /> + +
+ Workspace +

Claude Code sessions started inside this folder bind to the agent. The full path stays on this machine; the world shows only the folder's name.

+ {arrivals.map((a) => ( + + ))} + + {source.kind === 'path' && ( + setPath(e.target.value)} + /> + )} +
+ +
+
+

+ Connect Claude Code +

+ {ingest?.claudeCode ? ( + + ingest ready + + ) : ingest ? ( + ingest turned off + ) : ( + ingest not reported + )} +
+

The Runloop plugin's hooks report each session (tool calls, results, subagents). Runloop records what they report; it never runs or approves the agent's tools. Install it once:

+ +
+ + {error && ( +

+ + {error} +

+ )} + +
+ + +
+
+ ); +} diff --git a/apps/studio/src/components/agents/AddedAgentStatus.tsx b/apps/studio/src/components/agents/AddedAgentStatus.tsx new file mode 100644 index 0000000..ba7fcc9 --- /dev/null +++ b/apps/studio/src/components/agents/AddedAgentStatus.tsx @@ -0,0 +1,84 @@ +'use client'; +/** + * Add to World's last step (contract §4.3): what the new agent is doing now, + * read from the live world, never from the create answer once that is stale. + * Until a real hook arrives the building stands unpainted and this says how + * to connect Claude Code; from the first real event it shows the agent's + * honest health (icon + text), its attention line and when the first event + * came. `onArrived` runs once when that first event lands while it is shown. + */ +import type { AgentSummary } from '@runloop-world/events'; +import { useEffect, useRef, type ReactNode } from 'react'; +import { CC } from '../../lib/claude-code'; +import { useTicker } from '../../state/hooks'; +import { useWorld } from '../../world/context'; +import { clockParts } from '../../world/interpolate'; +import { addedAgentView } from '../../world/selectors'; +import { CommandSnippet } from '../../ui/CommandSnippet'; +import { Icon } from '../../ui/Icon'; +import { whenText } from '../world/hud/bits'; +import styles from './agents.module.css'; + +export default function AddedAgentStatus({ + created, + onArrived, + more, + titleClass = styles.flowTitle, +}: { + created: AgentSummary; + onArrived?: (agentId: string) => void; + /** Extra help under the install command while it waits (the dialog links to Connect). */ + more?: ReactNode; + titleClass?: string | undefined; +}) { + useWorld((s) => s.rev); + const presence = useWorld((s) => s.snapshot?.agents.find((a) => a.agentId === created.agentId) ?? null); + const summary = useWorld((s) => s.agents.find((a) => a.agentId === created.agentId) ?? null); + const log = useWorld((s) => s.log); + const today = useWorld((s) => (s.clock ? clockParts(s.clock.tick).day : 0)); + useTicker(true, 1000); + const v = addedAgentView(created, presence, summary, log, Date.now()); + + // The first real event while this step is open: hand over (once) to the agent's own card. + const waited = useRef(v.waiting); + const arrived = useRef(onArrived); + arrived.current = onArrived; + useEffect(() => { + if (waited.current && !v.waiting) arrived.current?.(created.agentId); + waited.current = v.waiting; + }, [v.waiting, created.agentId]); + + if (v.waiting) { + return ( + <> +

+ + {v.name} is in the Agent District, waiting for its first event. +

+

+ The building stands unpainted until Claude Code really sends a hook from {created.workspace ? `“${created.workspace.label}”` : 'its workspace'}. Install the plugin once, then run claude in that folder: +

+ +

{more ?? <>Or load it from a Runloop checkout with {CC.pluginDir}.}

+ + ); + } + return ( + <> +

+ + {v.name} · {v.health.text} +

+

+ Real · Claude Code · observed + {v.firstEventTick !== null ? ` · first event ${whenText(v.firstEventTick, today)}` : ''} · {v.lastEvent} +

+ {v.attention && ( +

+ + {v.attention} +

+ )} + + ); +} diff --git a/apps/studio/src/components/agents/AgentsPanel.tsx b/apps/studio/src/components/agents/AgentsPanel.tsx new file mode 100644 index 0000000..b87f15a --- /dev/null +++ b/apps/studio/src/components/agents/AgentsPanel.tsx @@ -0,0 +1,190 @@ +'use client'; +import type { ArrivalSummary } from '@runloop-world/events'; +import { useState } from 'react'; +import { formatWhen } from '../../state/format'; +import { useTicker } from '../../state/hooks'; +import { claimSession, WorldApiError } from '../../world/api'; +import { refreshAgents } from '../../world/connect'; +import { useWorld, useWorldStore } from '../../world/context'; +import { healthView, lastEventText } from '../../world/selectors'; +import { Button } from '../../ui/Button'; +import { Icon } from '../../ui/Icon'; +import { toast } from '../../ui/Toast'; +import styles from './agents.module.css'; + +/** + * Agents mode (plan §5): the Agent District's real agents with their + * connection, and Arrivals: Claude Code sessions no agent's workspace + * contains yet, which can be claimed (their earlier activity is backfilled) + * or used to found a new agent. + */ +export default function AgentsPanel({ onAddToWorld }: { onAddToWorld: (fromSessionId?: string) => void }) { + const store = useWorldStore(); + useWorld((s) => s.rev); + const presences = useWorld((s) => s.snapshot?.agents); + const summaries = useWorld((s) => s.agents); + const arrivals = useWorld((s) => s.arrivals); + const api = useWorld((s) => s.agentsApi); + const town = useWorld((s) => s.town); + useTicker(true, 5000); + const now = Date.now(); + const list = presences ?? []; + const free = (town?.agentLots.length ?? 0) - list.length; + const unhoused = summaries.filter((a) => !a.housed); + + return ( +
+
+
+

Agents

+ +
+

Real agents live in the Agent District. A building is their place in the town; the connection is the real integration, and the work happens where the agent runs.

+
+ + {api === 'unavailable' && ( +

+ + This runtime does not serve the agents API yet, so workspaces, sessions and Arrivals are not reported. Buildings below come from the world log. +

+ )} + {api === 'offline' && ( +

+ + Could not reach the agents API. Retrying every few seconds. +

+ )} + +
+

+ In the Agent District ({list.length}) · {free} free {free === 1 ? 'lot' : 'lots'} +

+ {list.length === 0 ? ( +
+

No real agent lives here yet. The town runs on its own; adding your Claude Code agent gives it a building next to the residents.

+ +
+ ) : ( +
    + {list.map((a) => { + const h = healthView(a.health); + const summary = summaries.find((s) => s.agentId === a.agentId); + return ( +
  • + +
    • + ); + })} +
+ )} + {unhoused.length > 0 && ( +

+ {unhoused.length} {unhoused.length === 1 ? 'agent is' : 'agents are'} registered but not housed yet; Runloop retries at start-up. +

+ )} +
+ +
+

+ Arrivals ({arrivals.length}) +

+

Claude Code sessions whose folder no agent owns. They are traced as usual; claim one to bring its activity into a building.

+ {arrivals.length === 0 ? ( +

{api === 'ok' ? 'None right now.' : 'Not reported.'}

+ ) : ( +
    + {arrivals.map((a) => ( + onAddToWorld(a.sessionId)} /> + ))} +
+ )} +
+ +

+ {' '} + ·{' '} + +

+
+ ); +} + +function Arrival({ arrival, onFound }: { arrival: ArrivalSummary; onFound: () => void }) { + const store = useWorldStore(); + const agents = useWorld((s) => s.snapshot?.agents); + const [target, setTarget] = useState(''); + const [busy, setBusy] = useState(false); + const [error, setError] = useState(null); + const claim = async () => { + if (!target) return; + setBusy(true); + setError(null); + try { + const res = await claimSession(target, arrival.sessionId); + toast({ tone: 'ok', title: `Claimed for ${res.agent.displayName}`, description: `${res.backfilledRuns} ${res.backfilledRuns === 1 ? 'run' : 'runs'} of earlier activity replayed into the building.` }); + void refreshAgents(store); + } catch (e) { + setError(e instanceof WorldApiError ? e.message : e instanceof Error ? e.message : String(e)); + } finally { + setBusy(false); + } + }; + return ( +
  • +
    + + {arrival.projectLabel} + session {arrival.sessionId.slice(0, 8)} +
    + + {arrival.pathHint ?? 'folder not known since the last restart'} · {arrival.runIds.length} {arrival.runIds.length === 1 ? 'run' : 'runs'} · first seen {formatWhen(arrival.firstSeenAt)} + +
    + {(agents?.length ?? 0) > 0 && ( + <> + + + + )} + +
    + {error && ( +

    + + {error}. Nothing changed. +

    + )} +
    • + ); +} diff --git a/apps/studio/src/components/agents/AssignTask.module.css b/apps/studio/src/components/agents/AssignTask.module.css new file mode 100644 index 0000000..1c4c07b --- /dev/null +++ b/apps/studio/src/components/agents/AssignTask.module.css @@ -0,0 +1,300 @@ +/* Assign task: the agent panel's Tasks section and the dispatch dialog. */ + +.section { + display: flex; + flex-direction: column; + gap: var(--rl-space-2); +} +.sectionHead { + display: flex; + align-items: baseline; + justify-content: space-between; + gap: var(--rl-space-2); +} +.h3 { + color: var(--rl-text-faint); + font-size: var(--rl-text-2xs); + font-weight: 600; + letter-spacing: 0.06em; + text-transform: uppercase; +} +.note { + display: inline-flex; + flex-wrap: wrap; + align-items: center; + gap: 4px; + color: var(--rl-text-muted); + font-size: var(--rl-text-xs); + line-height: 1.45; +} +.muted { + color: var(--rl-text-muted); +} +.mono { + font-family: var(--rl-font-mono); + font-size: var(--rl-text-xs); +} +.card { + display: flex; + flex-direction: column; + gap: 6px; + padding: var(--rl-space-3) var(--rl-space-4); + border-radius: var(--rl-radius-card); + background: var(--rl-surface); +} +.actions { + display: flex; + flex-wrap: wrap; + gap: var(--rl-space-2); +} +.status { + display: inline-flex; + align-items: center; + gap: 6px; + font-weight: 600; + font-size: var(--rl-text-md); +} +.status[data-tone='live'] { + color: var(--rl-accent-text); +} +.status[data-tone='ok'] { + color: var(--rl-success); +} +.status[data-tone='warn'] { + color: var(--rl-warning); +} +.status[data-tone='err'] { + color: var(--rl-failure); +} +.status[data-tone='muted'] { + color: var(--rl-text-muted); +} +.task { + overflow: hidden; + color: var(--rl-text); + font-size: var(--rl-text-sm); + text-overflow: ellipsis; + white-space: nowrap; +} +.links { + display: flex; + flex-wrap: wrap; + gap: 4px; + align-items: center; +} +.linkButton { + display: inline-flex; + align-items: center; + height: 24px; + padding: 0 8px; + border: 0; + border-radius: var(--rl-radius-sm); + background: var(--rl-surface-2); + color: var(--rl-text); + font: inherit; + font-size: var(--rl-text-xs); + font-weight: 500; + cursor: pointer; +} +.linkButton:hover { + background: var(--rl-surface-3); +} +.linkButton:focus-visible { + box-shadow: var(--rl-focus-ring); + outline: none; +} + +/* ── Dialog ── */ +.dialog { + width: min(640px, 100%); +} +.textarea { + min-height: 88px; + padding: var(--rl-space-2) var(--rl-space-3); + border: 0; + border-radius: var(--rl-radius-control); + background: var(--rl-surface-2); + font: inherit; + font-size: var(--rl-text-md); + line-height: 1.45; + resize: vertical; +} +.textarea[aria-invalid='true'] { + box-shadow: 0 0 0 1px var(--rl-failure); +} +.legend { + padding: 0; + margin: var(--rl-space-2) 0; + color: var(--rl-text-muted); + font-size: var(--rl-text-xs); + font-weight: 500; +} +.tools { + margin-top: 2px; + font-family: var(--rl-font-mono); + font-size: var(--rl-text-2xs); + color: var(--rl-text-muted); + white-space: pre-wrap; +} +.row3 { + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + gap: var(--rl-space-3); +} +.field { + display: flex; + flex-direction: column; + gap: 4px; + min-width: 0; +} +.field input, +.field select { + width: 100%; +} +.command { + display: flex; + gap: var(--rl-space-2); + align-items: flex-start; + padding: var(--rl-space-2) var(--rl-space-2) var(--rl-space-2) var(--rl-space-3); + border-radius: var(--rl-radius-control); + background: var(--rl-surface-2); +} +.command pre { + flex: 1; + min-width: 0; + margin: 0; + padding-top: 3px; + font-family: var(--rl-font-mono); + font-size: var(--rl-text-xs); + line-height: 1.55; + white-space: pre-wrap; + overflow-wrap: anywhere; +} +.command pre span { + color: var(--rl-text-faint); +} +.kv { + display: grid; + grid-template-columns: 110px minmax(0, 1fr); + gap: 6px var(--rl-space-3); + margin: 0; + font-size: var(--rl-text-sm); +} +.kv dt { + color: var(--rl-text-muted); +} +.kv dd { + min-width: 0; + margin: 0; + overflow-wrap: anywhere; +} +.chips { + display: flex; + flex-wrap: wrap; + gap: 6px; +} +.facts { + display: flex; + flex-direction: column; + gap: 6px; + padding: var(--rl-space-3); + border-radius: var(--rl-radius-control); + background: var(--rl-surface); + color: var(--rl-text-muted); + font-size: var(--rl-text-sm); + line-height: 1.45; +} +.facts li { + display: flex; + gap: var(--rl-space-2); +} +.facts li svg { + flex: none; + margin-top: 3px; +} +.facts strong { + color: var(--rl-text); + font-weight: 600; +} +.facts li[data-warn] svg, +.facts li[data-warn] strong { + color: var(--rl-warning); +} +.result { + display: flex; + flex-direction: column; + gap: 6px; + padding: var(--rl-space-3) var(--rl-space-4); + border-radius: var(--rl-radius-card); + background: var(--rl-surface); +} +.quote { + margin: 0; + padding: var(--rl-space-2) var(--rl-space-3); + border-left: 2px solid var(--rl-line); + color: var(--rl-text); + font-size: var(--rl-text-sm); + white-space: pre-wrap; + overflow-wrap: anywhere; +} +.output { + display: flex; + flex-direction: column; + gap: 2px; + max-height: 240px; + min-height: 72px; + margin: 0; + padding: var(--rl-space-2) var(--rl-space-3); + overflow-y: auto; + border-radius: var(--rl-radius-control); + background: var(--rl-surface-2); + font-family: var(--rl-font-mono); + font-size: var(--rl-text-xs); + line-height: 1.5; + list-style: none; +} +.output li { + display: flex; + gap: 6px; + overflow-wrap: anywhere; +} +.output li svg { + flex: none; + margin-top: 3px; +} +.output li[data-kind='system'], +.output li[data-kind='raw'] { + color: var(--rl-text-muted); +} +.output li[data-kind='tool'] { + color: var(--rl-text); +} +.output li[data-kind='result'] { + font-weight: 600; +} +.output li[data-kind='stderr'] { + color: var(--rl-warning); +} +.confirm { + display: flex; + flex-direction: column; + gap: var(--rl-space-2); + padding: var(--rl-space-3); + border-radius: var(--rl-radius-control); + background: var(--rl-surface); + box-shadow: inset 0 0 0 1px var(--rl-failure); + font-size: var(--rl-text-sm); +} +@media (max-width: 560px) { + .row3 { + grid-template-columns: 1fr; + } + .kv { + grid-template-columns: 1fr; + } +} + +/* The approval card over the town: a light scrim, so the agent's building stays in view (contract §4.1). */ +:global(div[data-placement]):has(> .approval) { + background: rgb(6 7 9 / 0.3); +} diff --git a/apps/studio/src/components/agents/AssignTask.tsx b/apps/studio/src/components/agents/AssignTask.tsx new file mode 100644 index 0000000..108aabd --- /dev/null +++ b/apps/studio/src/components/agents/AssignTask.tsx @@ -0,0 +1,673 @@ +'use client'; +/** + * Assign task (plan §4.4, §5): a real agent's headless Claude Code run, + * approved by the user. Compose → review the exact command (model, budget, + * tools, workspace, environment names) → approve → follow its output → the + * outcome as Claude Code reported it ("Claude Code reports done"). + * + * Nothing runs until a person approves it on the runtime's own approval page + * (`preview.approveUrl`, opened in a window): no web page, Studio included, + * can approve a task by itself. Studio keeps showing the exact command and + * follows the approval (GET /api/dispatches/previews/:id). The approval is + * single use and expires after 5 minutes. + */ +import { + DISPATCH_COPY, + DISPATCH_PRESETS, + SSE_END_EVENT, + type AgentPresence, + type DispatchOutputFrame, + type DispatchPolicyResponse, + type DispatchPreset, + type DispatchPreview, + type DispatchSummary, +} from '@runloop-world/events'; +import { useEffect, useId, useMemo, useRef, useState } from 'react'; +import { formatWhen } from '../../state/format'; +import { refreshAgents } from '../../world/connect'; +import { useWorldStore } from '../../world/context'; +import { WorldApiError } from '../../world/api'; +import type { CapabilityView } from '../../world/selectors'; +import { Button } from '../../ui/Button'; +import { Chip } from '../../ui/Chip'; +import { Dialog } from '../../ui/Dialog'; +import { Icon } from '../../ui/Icon'; +import { useStudioNav } from '../overlays/nav'; +import { cancelDispatch, dispatchStreamUrl, getDispatch, getDispatchPolicy, getPreviewStatus, previewDispatch } from './AssignTaskApi'; +import { approvalOutcomeText, commandLine, costText, dispatchStatusView, expiresIn, outputLines, type DispatchStatusView } from './AssignTaskView'; +import base from './agents.module.css'; +import styles from './AssignTask.module.css'; + +const PRESET_TEXT: Record = { + read_only: { title: 'Read-only', text: 'Claude Code may read and search files. It cannot edit or run commands.' }, + edit_test: { title: 'Edit + test', text: 'Adds editing files and running pnpm test. Nothing else is allowed.' }, +}; + +const errorText = (e: unknown) => + e instanceof WorldApiError ? (e.message.endsWith('.') ? e.message : `${e.message}.`) : `Could not reach the runtime: ${e instanceof Error ? e.message : String(e)}.`; + +/** Status line with icon and words (never colour alone). */ +function StatusLine({ view }: { view: DispatchStatusView }) { + return ( + + + {view.title} + + ); +} + +/** + * The agent panel's task section: the latest dispatch (from the world's + * agent presence) and the Assign task button, disabled with its reason. + */ +export function AssignTaskSection({ + agentId, + agentName, + workspaceLabel, + submit, + latest, +}: { + agentId: string; + agentName: string; + workspaceLabel: string | null; + submit: CapabilityView | undefined; + latest: AgentPresence['dispatch']; +}) { + const [open, setOpen] = useState(false); + const [watch, setWatch] = useState(null); + const [summary, setSummary] = useState(null); + const nav = useStudioNav(); + const available = submit?.available ?? false; + + // The world says which dispatch is the latest and its phase; details come from the API. + useEffect(() => { + if (!latest) return setSummary(null); + let live = true; + getDispatch(latest.dispatchId) + .then((r) => live && setSummary(r.dispatch)) + .catch(() => live && setSummary(null)); + return () => { + live = false; + }; + }, [latest?.dispatchId, latest?.phase]); + + const view = summary ? dispatchStatusView(summary) : null; + return ( +
    +
    +

    + Tasks +

    + Launched by Runloop · traced +
    + {summary && view && ( +
    + + + “{summary.taskPreview}” + + + {formatWhen(summary.startedAt)} · {summary.model} · {costText(summary.reportedCostUsd)} + + + + {summary.runId && ( + + )} + +
    + )} +
    + +
    + {!available &&

    {submit?.reason ? `${submit.reason}.` : 'This runtime does not report whether it can dispatch tasks.'}

    } + setOpen(false)} agentId={agentId} agentName={agentName} workspaceLabel={workspaceLabel} watch={watch} /> +
    + ); +} + +type Step = 'compose' | 'review' | 'run'; + +export default function AssignTaskDialog({ + open, + onClose, + agentId, + agentName, + workspaceLabel, + watch, +}: { + open: boolean; + onClose: () => void; + agentId: string; + agentName: string; + workspaceLabel: string | null; + /** Open on this dispatch's progress instead of a new task. */ + watch?: string | null; +}) { + const id = useId(); + const store = useWorldStore(); + const nav = useStudioNav(); + const taskRef = useRef(null); + const [step, setStep] = useState('compose'); + const [policy, setPolicy] = useState(null); + const [task, setTask] = useState(''); + const [preset, setPreset] = useState('read_only'); + const [model, setModel] = useState('haiku'); + const [budget, setBudget] = useState('0.25'); + const [timeout, setTimeoutMin] = useState('10'); + const [preview, setPreview] = useState(null); + const [dispatchId, setDispatchId] = useState(null); + const [summary, setSummary] = useState(null); + const [frames, setFrames] = useState([]); + const [busy, setBusy] = useState(false); + /** The runtime's approval page is open; Studio waits for its outcome. */ + const [waiting, setWaiting] = useState(false); + const [error, setError] = useState(null); + const [confirmCancel, setConfirmCancel] = useState(false); + const [now, setNow] = useState(() => Date.now()); + + // Reset on open; load the policy (defaults, allowed models, caps) and the CLI probe. + useEffect(() => { + if (!open) return; + setError(null); + setBusy(false); + setWaiting(false); + setConfirmCancel(false); + setPreview(null); + setFrames([]); + setSummary(null); + setDispatchId(watch ?? null); + setStep(watch ? 'run' : 'compose'); + if (!watch) setTask(''); + getDispatchPolicy() + .then((p) => { + setPolicy(p); + if (!watch) { + setModel(p.policy.models[0] ?? 'haiku'); + setBudget(String(p.policy.defaultBudgetUsd)); + setTimeoutMin(String(p.policy.defaultTimeoutMin)); + } + }) + .catch((e: unknown) => setError(e instanceof WorldApiError && e.code === 'dispatch_disabled' ? 'Task dispatch is off in this runtime.' : errorText(e))); + }, [open, watch]); + + // The approval countdown. + useEffect(() => { + if (!open || step !== 'review') return; + const t = setInterval(() => setNow(Date.now()), 1000); + return () => clearInterval(t); + }, [open, step]); + + // Follow a dispatch: its output over SSE, its summary by polling (stopping, run id, outcome). + useEffect(() => { + if (!open || step !== 'run' || !dispatchId) return; + let live = true; + const refresh = () => + getDispatch(dispatchId) + .then((r) => { + if (!live) return; + setSummary(r.dispatch); + if (r.dispatch.status === 'finished') void refreshAgents(store); + }) + .catch((e: unknown) => live && setError(errorText(e))); + void refresh(); + const poll = setInterval(refresh, 2000); + const es = new EventSource(dispatchStreamUrl(dispatchId)); + es.onmessage = (m) => { + try { + const f = JSON.parse(m.data as string) as DispatchOutputFrame; + setFrames((prev) => [...prev, f]); + } catch { + // ignore a frame that does not parse + } + }; + es.addEventListener(SSE_END_EVENT, () => { + es.close(); + void refresh(); + }); + return () => { + live = false; + clearInterval(poll); + es.close(); + }; + }, [open, step, dispatchId, store]); + + const budgetNum = Number(budget); + const timeoutNum = Number(timeout); + const caps = policy?.policy; + const budgetOk = Number.isFinite(budgetNum) && budgetNum >= 0.01 && budgetNum <= (caps?.maxBudgetUsd ?? 1); + const timeoutOk = Number.isInteger(timeoutNum) && timeoutNum >= 1 && timeoutNum <= (caps?.maxTimeoutMin ?? 30); + const taskOk = task.trim().length > 0 && !task.trim().startsWith('-'); + const cliOk = policy?.cli.state === 'ok'; + const canPreview = !!caps?.enabled && cliOk && taskOk && budgetOk && timeoutOk && !busy; + + const doPreview = async () => { + setBusy(true); + setError(null); + try { + const r = await previewDispatch(agentId, { task: task.trim(), preset, model, maxBudgetUsd: budgetNum, timeoutMin: timeoutNum }); + setPreview(r.preview); + setNow(Date.now()); + setStep('review'); + } catch (e) { + setError(errorText(e)); + } finally { + setBusy(false); + } + }; + + /** A person approves on the runtime's own page; Studio only opens it. */ + const doApprove = () => { + if (!preview) return; + setError(null); + window.open(preview.approveUrl, 'runloop-approve', 'popup,width=720,height=860'); + setWaiting(true); + }; + + // Follow the approval page's outcome. Its window is cross-origin isolated (COOP), so poll the + // runtime rather than watch the window. + useEffect(() => { + if (!open || !waiting || !preview) return; + let live = true; + const check = () => + getPreviewStatus(preview.previewId).then( + (st) => { + if (!live || st.state === 'awaiting_approval') return; + setWaiting(false); + if (st.state === 'approved' && st.dispatchId) { + setDispatchId(st.dispatchId); + setStep('run'); + return; + } + const why = approvalOutcomeText(st); + setError(st.state === 'approved' ? why : `${why} Nothing was run.`); + setPreview(null); + setStep('compose'); + }, + (e: unknown) => live && setError(errorText(e)), + ); + void check(); + const t = setInterval(() => void check(), 1000); + return () => { + live = false; + clearInterval(t); + }; + }, [open, waiting, preview]); + + const doCancel = async () => { + if (!dispatchId) return; + setBusy(true); + try { + setSummary((await cancelDispatch(dispatchId)).dispatch); + setConfirmCancel(false); + } catch (e) { + setError(errorText(e)); + } finally { + setBusy(false); + } + }; + + const lines = useMemo(() => outputLines(frames), [frames]); + const outputRef = useRef(null); + useEffect(() => { + const el = outputRef.current; + if (el) el.scrollTop = el.scrollHeight; + }, [lines.length]); + const view = summary ? dispatchStatusView(summary) : null; + const where = workspaceLabel ? `“${workspaceLabel}”` : 'its workspace'; + + return ( + +
    { + e.preventDefault(); + if (step === 'compose' && canPreview) void doPreview(); + else if (step === 'review' && preview) void doApprove(); + }} + > +
    +

    + {step === 'run' ? 'Task' : `Assign a task to ${agentName}`} +

    +

    + {step === 'run' + ? `Claude Code, headless, in ${where}. Its hooks trace the session like any other.` + : `Claude Code runs headless in ${where}. Nothing runs until you approve the exact command on Runloop’s own approval page.`} +

    +
    + +
    + {policy && !cliOk && step !== 'run' && ( +

    + + Claude Code cannot be run: {policy.cli.reason ?? 'not checked yet'}. +

    + )} + {caps && !caps.enabled && step !== 'run' && ( +

    + + Your organisation's policy does not allow task dispatch. +

    + )} + + {step === 'compose' && ( + <> + +