diff --git a/portable/README.md b/portable/README.md index 80faf03..3f02c30 100644 --- a/portable/README.md +++ b/portable/README.md @@ -1,11 +1,17 @@ -# Portable sessions +# Portable agent workspace -This directory is reserved for running Abalone from a portable live system such -as [Tails](https://tails.net/about/index.en.html). Nothing here is supported -yet: USB/Tails is unqualified -([getting started](../docs/getting-started.md#supported-system)), and the NEAR -route fails on Tails because NVIDIA's attestation service refuses Tor exits -(issue [#15](https://github.com/amiller/abalone/issues/15)). +Experimental components for booting into a local agent workspace, with ARM64 +rehearsals before Raspberry Pi USB testing. The presentation is independent of +the existing launcher. -A Tails-based portable session is being researched. Results that hold up will -land here as public instructions and checks. +- [Workspace](workspace/README.md): retained Linux terminals and a bounded Pi SDK + lifecycle adapter. Its local model is scripted; SDK tool execution is real. +- [ARM64 rehearsal](raspberry-pi/README.md): disposable QEMU guest and kiosk setup. +- [SDK runtime bundle](raspberry-pi/sdk-runtime.md): package integrity checks and + on-target runtime verification. + +Qualification is limited to the documented prototype checks. The USB image, +Pi hardware boot, Tor-only startup, live inference and funding, and encrypted +reboot recovery remain development milestones. Use disposable state and the +component instructions. [Supported-system scope](../docs/getting-started.md#supported-system) +continues to apply. diff --git a/portable/raspberry-pi/README.md b/portable/raspberry-pi/README.md new file mode 100644 index 0000000..020b796 --- /dev/null +++ b/portable/raspberry-pi/README.md @@ -0,0 +1,103 @@ +# ARM64 portable rehearsal + +Boot a disposable ARM64 development guest to validate the portable filesystem and +startup services before testing a separate Raspberry Pi USB image. This is a VM +boot spike: it does not yet contain the workspace UI, agent, payment integration, +or a Tor-only network boundary. User-mode networking permits ordinary egress. +Use synthetic work and no payment credentials. + +The initial host recipe targets Ubuntu 22.04 x86_64 with `qemu-img`, `genisoimage`, +`curl`, `ssh-keygen`, `dpkg-deb`, and the shared libraries required by QEMU already +installed. ARM QEMU and EFI packages are extracted into a user-owned directory; +no system package install or existing VM changes are needed. The package versions +are pinned; if the host's authenticated apt indexes no longer offer them, review +and update the pins instead of silently substituting a different version. + +Choose dedicated storage on a disk with room for the image and a sparse 12 GB +writable overlay. Heavy work belongs on a build host, not a small laptop. + +```sh +export PORTABLE_REHEARSAL_DIR=/path/to/dedicated/portable-rehearsal +./portable/raspberry-pi/rehearse.sh tools +./portable/raspberry-pi/rehearse.sh image +./portable/raspberry-pi/rehearse.sh prepare +./portable/raspberry-pi/rehearse.sh run +``` + +Run the final command in a persistent terminal. In another terminal on that host: + +```sh +export PORTABLE_REHEARSAL_DIR=/path/to/dedicated/portable-rehearsal +./portable/raspberry-pi/rehearse.sh ssh 'uname -m; cloud-init status --wait' +``` + +SSH is bound to host loopback port 2222 and VNC to loopback port 5901. Forward them +through SSH to inspect a remote guest. No guest password is installed; the +rehearsal generates a dedicated key in `seed/`. The guest user has passwordless +sudo inside this disposable VM. Never reuse the seed or key for a shipped image. +The base image has no graphical desktop; VNC is wired for a future workspace. + +The pinned base is Debian's official +[Bookworm ARM64 generic image](https://cloud.debian.org/images/cloud/bookworm/20261006-2623/). +The script verifies its pinned SHA-512 before accepting a download or preparing +an overlay. Provenance and serial output go in `evidence/`; preserve those with +source revision and outcomes. The checksum was obtained from the publisher over +HTTPS; detached-signature verification is not implemented by this spike. + +QEMU's [generic virt board](https://www.qemu.org/docs/master/system/arm/virt.html) +runs with four Cortex-A72 CPUs, 4 GB RAM, and software TCG emulation on x86. Its +firmware and virtio devices differ from a Pi. Share filesystem contents and +service definitions with the eventual Pi package, but provide machine-specific +kernel/firmware configuration. Emulated boot time does not predict Pi timing. + +## Acceptance loop + +1. Preserve the immutable base image; use one writable overlay per rehearsal. +2. Record serial `PORTABLE_ARM64_READY`, `uname -m`, cloud-init status, image hash, + source revision, and the actual checks run. These establish a booted guest, + not a working agent experience. +3. As workspace integration lands, exercise local interaction before networking, + a real synthetic agent task, connection loss, and restart recovery. +4. Once Tor enforcement lands, independently capture egress and test failure + closure. A Tor status label or this VM's NAT configuration is not evidence. +5. Shut down with `rehearse.sh ssh 'sudo poweroff'`. Retain logs and results; + remove disposable images, keys, and build outputs after collecting evidence. + A new `prepare` refuses to overwrite an existing overlay. + +The hardware gate remains a separate USB boot on the target Pi, covering firmware, +display, input, USB compatibility, and three measured cold boots. Do not flash a +system disk or disturb a running Pi as part of the VM rehearsal. + +## Workspace and kiosk in the disposable guest + +Copy `portable/workspace/` and `portable/raspberry-pi/provision-guest.sh` into the +rehearsal user's home directory, preserving the directory layout, then run: + +```sh +bash portable/raspberry-pi/provision-guest.sh --kiosk +``` + +This guest-only command verifies a pinned official Node 24 ARM64 archive, installs +it under `~/.local/opt`, and enables `portable-workspace.service` before installing +optional Chromium, Xorg, Openbox, and fonts from Debian's authenticated package +repositories. Omit `--kiosk` for backend-only iteration. Kiosk packages use the +configured Debian repository versions; record installed versions when qualifying +an image. This step is not a reproducible final image build. + +The kiosk uses tty1 autologin as the disposable rehearsal user and opens the +loopback workspace. This convenience account has sudo; it is not the final +agent authority boundary. Do not add secrets. A graphical failure should leave +SSH available for diagnosis. Inspect `journalctl -u portable-workspace` and the +user's Xorg log. VNC displays the guest; it is not a web server port. + +Copy [verify-workspace.py](verify-workspace.py) into the guest and run it with +Python 3 to exercise a real terminal via the HTTP API. It checks `uname -m` +through the PTY and closes its session. Retain its JSON output as evidence. + +## SDK runtime bundle + +[Package and verify the SDK runtime](sdk-runtime.md) separately from guest +provisioning. The bundle verifier exercises real SDK tools, native clipboard +binding loading and QuickJS on the machine where it runs. Host results do not +qualify another architecture; run the verifier inside the ARM guest before +connecting its workspace to the bundle. This does not perform model inference. diff --git a/portable/raspberry-pi/package-sdk.mjs b/portable/raspberry-pi/package-sdk.mjs new file mode 100644 index 0000000..5983db5 --- /dev/null +++ b/portable/raspberry-pi/package-sdk.mjs @@ -0,0 +1,1468 @@ +#!/usr/bin/env node +/** + * Build a small, transferable runtime bundle of the vendored Pi SDK + * (@earendil-works/pi-coding-agent 1.1.x) for deployment on another machine, + * by default a Raspberry Pi class ARM64 (aarch64) Linux guest. + * + * Safety contract (fail closed; this tool never deletes anything): + * - The output directory must NOT exist, or must be an empty real directory. + * A nonempty output, a symlinked output (or symlinked ancestor), or a + * preexisting archive/checksum/summary target is refused. + * - The output must be disjoint from the source tree: it may not lie inside + * the source, and the source may not lie inside it (paths canonicalized). + * - Older bundles and sibling archives are NEVER pruned. Retire outputs by + * hand. + * + * Bundle format (simple and portable): + * - Regular files and directories only. NO symlinks anywhere: any symlink + * found while copying is a hard error. + * - manifest.txt holds the sha256 of EVERY file in the bundle except + * manifest.txt itself (it cannot hash itself). manifest.json (metadata), + * THIRD_PARTY_NOTICES.md and the marker are covered like any other file. + * - manifest.txt is the integrity root of the bundle. It is integrity + * relative to the separately trusted transfer digest (the .tar.gz.sha256 + * sidecar verified over the trusted transfer channel); it is NOT + * authenticity of the source tree. + * + * Closure policy (static scan, classified, fail closed on the unexplained): + * - Literal import/require/require.resolve/import() specifiers are traced. + * - Missing RELATIVE imports are errors for first-party code and must be + * classified for third-party code (OPTIONAL_IMPORTS). + * - Unresolved third-party bare imports are failures unless classified in + * OPTIONAL_IMPORTS with a concrete audited reason. + * - Nonliteral (computed) require()/import() sites must belong to a package + * listed in DYNAMIC_IMPORT_AUDIT with an audited reason. + * This is a heuristic static scan, not a complete static closure; runtime + * verification covers only the exercised paths (SDK import, quickjs, + * write tool). No universal SDK completeness is claimed. + * + * Native policy: + * - Every .node ELF must be target-arch, or host-arch WITH a target-arch + * sibling in the same package. Wrong-arch-only packages are hard failures. + * - WASM is recorded as portable. Host verification loads native bindings + * only for the host architecture; target-arch bindings are an explicit + * open gate for on-target verification (never claimed from the host). + * + * Provenance: the checkout may not be a git repo; the manifest records source + * and lockfile digests, exact package versions, license declarations and the + * generator script digests. Declared licenses are not notice provenance; the + * shipped license/notice files are preserved inside the packages. + * + * Usage: + * node package-sdk.mjs [--source ] [--out ] [--target-arch arm64|x64] + * [--target-platform linux] [--skip-verify] [--no-tar] [--help] + * + * Defaults are derived from this script's location: + * source: /vendor/pi (repo = the abalone checkout) + * out: $PI_SDK_BUNDLE_OUT or /../tmp/pi-sdk-- + * + * The tool performs no network access, no installs and no guest interaction. + * It only reads the existing built tree and writes to --out (plus the archive, + * checksum sidecar and build summary next to it). + */ +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { + chmodSync, + copyFileSync, + existsSync, + lstatSync, + mkdirSync, + readFileSync, + readdirSync, + readlinkSync, + realpathSync, + statSync, + writeFileSync, +} from "node:fs"; +import { isBuiltin } from "node:module"; +import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; +import { fileURLToPath } from "node:url"; + +const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url)); +const REPO_ROOT = resolve(SCRIPT_DIR, "..", ".."); +const BUNDLE_MARKER = ".pi-sdk-bundle"; + +class PackError extends Error {} + +function fail(message) { + throw new PackError(message); +} + +function log(step, message) { + console.error(`[package-sdk ${step}] ${message}`); +} + +function sha256File(file) { + return createHash("sha256").update(readFileSync(file)).digest("hex"); +} + +// --------------------------------------------------------------------------- +// Command line +// --------------------------------------------------------------------------- + +function parseArgs(argv) { + const options = { + source: resolve(REPO_ROOT, "vendor", "pi"), + out: process.env.PI_SDK_BUNDLE_OUT || "", + targetArch: "arm64", + targetPlatform: "linux", + skipVerify: false, + noTar: false, + help: false, + }; + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + const next = () => { + if (i + 1 >= argv.length) fail(`missing value for ${arg}`); + return argv[++i]; + }; + switch (arg) { + case "--source": + options.source = resolve(next()); + break; + case "--out": + options.out = resolve(next()); + break; + case "--target-arch": + options.targetArch = next(); + break; + case "--target-platform": + options.targetPlatform = next(); + break; + case "--skip-verify": + options.skipVerify = true; + break; + case "--no-tar": + options.noTar = true; + break; + case "--help": + case "-h": + options.help = true; + break; + default: + fail(`unknown argument: ${arg}`); + } + } + if (!["arm64", "x64"].includes(options.targetArch)) { + fail(`unsupported --target-arch: ${options.targetArch} (supported: arm64, x64)`); + } + if (options.targetPlatform !== "linux") { + fail(`unsupported --target-platform: ${options.targetPlatform} (only linux is wired up)`); + } + if (!options.out) { + options.out = resolve(REPO_ROOT, "..", "tmp", `pi-sdk-${options.targetPlatform}-${options.targetArch}`); + } + return options; +} + +// --------------------------------------------------------------------------- +// Output safety (nothing is ever deleted by this tool) +// --------------------------------------------------------------------------- + +/** + * Validate the output location before any write. Refuses: + * - an existing nonempty output directory + * - an existing output that is a file or symlink + * - symlinked ancestors (path redirection) + * - output inside the source tree, or the source inside the output + * - preexisting archive/checksum/summary targets next to the output + */ +function assertSafeOutput(source, outDir) { + if (!isAbsolute(outDir)) fail(`output path is not absolute after resolution: ${outDir}`); + // Preexisting archive/checksum/summary targets (any file type, incl. symlinks). + for (const suffix of [".tar.gz", ".tar.gz.sha256", ".build-summary.json"]) { + const target = `${outDir}${suffix}`; + let targetStat = null; + try { + targetStat = lstatSync(target); + } catch { + // absent + } + if (targetStat) { + fail(`refusing to touch preexisting output target: ${target} (remove it by hand if you are sure)`); + } + } + // The output itself. + const outStat = existsSync(outDir) ? lstatSync(outDir) : null; + if (outStat) { + if (outStat.isSymbolicLink()) fail(`output path is a symlink: ${outDir}`); + if (!outStat.isDirectory()) fail(`output path exists and is not a directory: ${outDir}`); + const entries = readdirSync(outDir); + if (entries.length > 0) { + fail( + `output directory is not empty (${entries.length} entries, first: ${entries[0]}): ${outDir}. ` + + "This tool never deletes existing output; choose a fresh directory or empty it by hand.", + ); + } + } + // No symlinked ancestors: every existing component must be a real directory. + let cursor = dirname(outDir); + const ancestors = []; + for (;;) { + ancestors.push(cursor); + if (cursor === dirname(cursor)) break; + cursor = dirname(cursor); + } + for (const ancestor of ancestors.reverse()) { + let st; + try { + st = lstatSync(ancestor); + } catch { + continue; // not created yet; parents are created with recursive mkdir below it + } + if (st.isSymbolicLink()) fail(`ancestor of the output is a symlink (path redirection): ${ancestor}`); + if (!st.isDirectory()) fail(`ancestor of the output is not a directory: ${ancestor}`); + } + // Disjointness, canonicalized against the deepest existing ancestor of out. + let probe = outDir; + while (!existsSync(probe)) { + const parent = dirname(probe); + if (parent === probe) break; + probe = parent; + } + const realAnchor = realpathSync(probe); + const canonicalOut = outDir === probe ? realAnchor : join(realAnchor, relative(probe, outDir)); + const realSource = realpathSync(source); + if (canonicalOut === realSource || canonicalOut.startsWith(realSource + sep)) { + fail(`output lies inside the source tree: ${outDir} (canonical ${canonicalOut}, source ${realSource})`); + } + if (realSource === canonicalOut || realSource.startsWith(canonicalOut + sep)) { + fail(`output is an ancestor of the source tree: ${outDir} (canonical ${canonicalOut}, source ${realSource})`); + } +} + +// --------------------------------------------------------------------------- +// Copy filters (third-party packages) +// --------------------------------------------------------------------------- + +// Directories that are always junk, at any depth. +const THIRD_PARTY_JUNK_DIRS = new Set([ + "__tests__", + "coverage", + ".nyc_output", + ".github", + ".git", + ".hg", + ".svn", + ".idea", + ".vscode", + ".husky", + ".circleci", + ".bin", + "test", + "tests", + "browser-test", + "system-test", + "bench", + "benchmarks", + ".history", // editor local-history snapshots (e.g. node-domexception ships stale test copies) +]); + +// Directories excluded only at the package root. Nested directories with these +// names can be runtime code (the yaml package ships dist/doc/, and some +// packages load examples or docs at runtime). +const THIRD_PARTY_ROOT_EXCLUDE_DIRS = new Set(["example", "examples", "docs", "doc"]); + +const THIRD_PARTY_DEV_FILE = + /^(\.|eslint|prettier|babel|webpack|rollup|vite\.config|vitest|jest|mocha|karma|nyc|tsconfig|ts-node|husky|lint-staged)/i; + +const THIRD_PARTY_TEST_FILE = /(\.(test|spec|bench)\.[cm]?js$)|(^test\.[cm]?js$)/i; + +const THIRD_PARTY_KEEP_BASENAME = /^(license|licence|copying|notice|readme|authors|contributors|changelog)/i; + +function thirdPartyKeep(entryName, isDir, depth = 0) { + if (isDir) { + if (THIRD_PARTY_JUNK_DIRS.has(entryName)) return false; + if (depth === 0 && THIRD_PARTY_ROOT_EXCLUDE_DIRS.has(entryName)) return false; + return true; + } + const ext = entryName.slice(entryName.lastIndexOf(".")); + if (entryName === "package.json" || entryName === "package-lock.json") return true; + if (ext === ".map" || ext === ".ts" || ext === ".tsx" || ext === ".mts" || ext === ".cts") return false; + if (THIRD_PARTY_TEST_FILE.test(entryName) || THIRD_PARTY_DEV_FILE.test(entryName)) return false; + if (ext === ".md") return THIRD_PARTY_KEEP_BASENAME.test(entryName); + if (ext === ".node" || ext === ".wasm" || ext === ".so") return true; + if (entryName.startsWith(".")) return false; + // Everything else (code, json, data files, images, html) is kept: dropping + // runtime assets the scanner cannot see risks a broken bundle. + return true; +} + +// --------------------------------------------------------------------------- +// Closure classification tables (audited on the real vendored tree, 2026-10-14) +// --------------------------------------------------------------------------- + +/** + * Unresolved third-party imports that are ACCEPTED, each with a concrete + * audited classification. Anything not listed here is a hard failure. + * Key: bare specifier root package name (or importer-qualified entries under + * "byImporter" for false positives inside generated-code templates). + */ +const OPTIONAL_IMPORTS = { + "@aws-sdk/signature-v4-crt": { + classification: "optional-native-dependency", + reason: "AWS CRT signer acceleration; only used when installed, runtime-guarded fallback exists in @aws-sdk/signature-v4-multi-region.", + }, + "@aws-sdk/signature-v4a": { + classification: "optional-dependency", + reason: "Asymmetric signature support package; optional peer of signature-v4-multi-region, guarded at runtime.", + }, + "supports-color": { + classification: "optional-dependency", + reason: "Color support probe of the debug package; optional by design.", + }, + kerberos: { + classification: "optional-native-dependency", + reason: "Native negotiate auth for proxy-agent-negotiate; only used when installed, plain negotiation otherwise.", + }, + bufferutil: { + classification: "optional-native-dependency", + reason: "Optional ws (WebSocket) permessage-deflate accelerator.", + }, + "utf-8-validate": { + classification: "optional-native-dependency", + reason: "Optional ws (WebSocket) UTF-8 validation accelerator.", + }, + byImporter: { + // String literals inside jiti's babel-compat generated-code templates, + // not real runtime imports of the SDK closure. + jiti: { + b: { + classification: "generator-template-false-positive", + reason: "String literal in jiti's babel compatibility templates; never executed as a real require by this bundle's paths.", + }, + "some-module": { + classification: "generator-template-false-positive", + reason: "Example string literal in jiti's generated-code templates; not a real dependency.", + }, + "./module": { + classification: "generator-template-false-positive", + reason: "Placeholder relative specifier inside jiti's bundled babel code-generation templates (dist/babel.cjs); not a real filesystem import.", + }, + "...": { + classification: "generator-template-false-positive", + reason: "Ellipsis placeholder string literal inside jiti's bundled babel templates (dist/babel.cjs); not a real specifier.", + }, + }, + }, +}; + +/** + * Packages that contain nonliteral (computed) require()/import() sites, with + * the audited reason the closure scanner accepts them. A nonliteral site in a + * package NOT listed here is a hard failure ("unaudited dynamic root"). + */ +const DYNAMIC_IMPORT_AUDIT = { + "@earendil-works/pi-ai": { + classification: "audited-computed-builtin-import", + reason: "Dynamic imports of node builtins via constant string concatenation (browser-bundler workaround: auth/context.js, env-api-keys.js, bedrock-converse-stream.lazy.js); no third-party runtime edges.", + }, + "@earendil-works/pi-coding-agent": { + classification: "audited-dynamic-root", + reason: "dist/core/extensions/loader.js imports user-specified extension paths at runtime. Extensions are disabled under verifier isolation (noExtensions) and extension loading is outside bundle acceptance.", + }, + "@silvia-odwyer/photon-node": { + classification: "audited-computed-builtin-import", + reason: "Template-literal import of the node builtin 'util'; constant specifier.", + }, + jiti: { + classification: "audited-runtime-transpiler", + reason: "jiti is a runtime transpiler and requires arbitrary user-specified modules by design; only its fixed entry paths are used by this bundle's closure.", + }, + marked: { + classification: "audited-cli-only", + reason: "bin/main.js loads a user config file path at runtime; CLI-only, not on the SDK runtime path.", + }, + protobufjs: { + classification: "audited-internal-lazy-load", + reason: "dist/light build lazily requires its own internal parser/tokenizer modules by computed name.", + }, + yaml: { + classification: "audited-cli-only", + reason: "dist/cli.mjs loads visitor plugins by path; CLI-only, not on the SDK runtime path.", + }, +}; + +// --------------------------------------------------------------------------- +// Import scanning +// --------------------------------------------------------------------------- + +const SPECIFIER_INVALID_CHARS = /[{}$()=,:;%<>[\]!&|+*?^'"\\\s]/; + +function plausibleSpecifier(spec) { + return spec.length > 0 && spec !== "." && spec !== ".." && !SPECIFIER_INVALID_CHARS.test(spec); +} + +const IMPORT_RE = + /(?:\bfrom\s*|\bimport\s*|\brequire\s*(?:\.resolve)?\s*\(\s*|\bimport\s*\(\s*)(["'])((?:[^"'\n\\]|\\.)*)\1/g; + +// require(...) / import(...) whose first argument is NOT a string literal. +const DYNAMIC_CALL_RE = /\b(?:require(?:\s*\.\s*resolve)?|import)\s*\(\s*([^)\n;]{0,80})/g; + +/** + * Remove // and /* *\/ comments while preserving string literals, so that + * prose like "the first grapheme from 'after'" in a comment cannot be mistaken + * for an import specifier. Regex literals (including ones containing quotes, + * e.g. .replace(/"/g, ...)) are copied verbatim so they cannot desync the + * string tracking; a bail-out on newline keeps malformed input from running + * away. Conservative: anything misparsed can only leave a string temporarily + * unbalanced, never add imports. + */ +const REGEX_PRECEDING_CHARS = "([{=,:;!&|?+-*%~^<>"; +const REGEX_PRECEDING_KEYWORDS = new Set([ + "return", + "typeof", + "instanceof", + "in", + "of", + "new", + "delete", + "void", + "throw", + "case", + "do", + "else", + "yield", + "await", +]); + +function lastSignificantToken(out) { + let end = out.length; + while (end > 0 && /\s/.test(out[end - 1])) end--; + let start = end; + while (start > 0 && /[A-Za-z0-9_$]/.test(out[start - 1])) start--; + if (start !== end) { + return { word: out.slice(start, end), char: null }; + } + return { word: null, char: end > 0 ? out[end - 1] : null }; +} + +function regexCanStartHere(out) { + const { word, char } = lastSignificantToken(out); + if (word !== null) return REGEX_PRECEDING_KEYWORDS.has(word); + if (char === null) return true; // start of file + return REGEX_PRECEDING_CHARS.includes(char); +} + +function stripComments(source) { + let out = ""; + let mode = "code"; + for (let i = 0; i < source.length; ) { + const char = source[i]; + const next = source[i + 1]; + if (mode === "code") { + if (char === "/" && next === "/") { + mode = "line"; + out += " "; + i += 2; + } else if (char === "/" && next === "*") { + mode = "block"; + out += " "; + i += 2; + } else if (char === "/" && regexCanStartHere(out)) { + mode = "regex"; + out += char; + i += 1; + } else { + if (char === '"') mode = "dquote"; + else if (char === "'") mode = "squote"; + else if (char === "`") mode = "template"; + out += char; + i += 1; + } + } else if (mode === "line") { + if (char === "\n") { + mode = "code"; + out += char; + } + i += 1; + } else if (mode === "block") { + if (char === "*" && next === "/") { + mode = "code"; + i += 2; + } else { + i += 1; + } + } else if (mode === "regex") { + // Copy regex literals verbatim; quotes inside them must not open strings. + if (char === "\n") { + // Malformed (unterminated) regex literal: bail out conservatively. + mode = "code"; + continue; + } + if (char === "\\") { + out += char + (next ?? ""); + i += 2; + continue; + } + if (char === "[") mode = "regexClass"; + else if (char === "/") mode = "code"; + out += char; + i += 1; + } else if (mode === "regexClass") { + if (char === "\\") { + out += char + (next ?? ""); + i += 2; + continue; + } + if (char === "]") mode = "regex"; + out += char; + i += 1; + } else { + // Inside a string literal: copy verbatim, honor escapes. + if (char === "\\") { + out += char + (next ?? ""); + i += 2; + continue; + } + if ((mode === "dquote" && char === '"') || (mode === "squote" && char === "'") || (mode === "template" && char === "`")) { + mode = "code"; + } + out += char; + i += 1; + } + } + return out; +} + +function scanFile(file) { + const specifiers = new Set(); + const dynamicSites = []; + let source; + try { + source = readFileSync(file, "utf8"); + } catch { + return { specifiers, dynamicSites }; + } + source = stripComments(source); + IMPORT_RE.lastIndex = 0; + let match; + while ((match = IMPORT_RE.exec(source)) !== null) { + const spec = match[2]; + if (!plausibleSpecifier(spec) || spec.startsWith("http:") || spec.startsWith("https:") || spec.startsWith("data:")) continue; + specifiers.add(spec); + } + DYNAMIC_CALL_RE.lastIndex = 0; + while ((match = DYNAMIC_CALL_RE.exec(source)) !== null) { + const arg = match[1].trim(); + if (!arg) continue; + // A quoted prefix is not necessarily a literal argument: concatenation + // must remain subject to the dynamic-import audit. + if (/^(?:"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*')\s*$/.test(arg)) continue; + if (arg.startsWith("`")) { + const end = arg.indexOf("`", 1); + const template = end > 0 ? arg.slice(1, end) : arg.slice(1); + if (end > 0 && arg.slice(end + 1).trim() === "" && !template.includes("${")) { + // Constant template literal: treat as a literal specifier. + if (plausibleSpecifier(template)) specifiers.add(template); + continue; + } + } + dynamicSites.push(arg.slice(0, 60)); + } + return { specifiers, dynamicSites }; +} + +function isBuiltinSpecifier(spec) { + if (spec.startsWith("node:") || spec.startsWith("bun:")) return true; + if (spec === "electron") return true; + try { + return isBuiltin(spec); + } catch { + return false; + } +} + +// --------------------------------------------------------------------------- +// Package resolution +// --------------------------------------------------------------------------- + +function readJson(file) { + return JSON.parse(readFileSync(file, "utf8")); +} + +function findPackageDir(fromDir, name) { + let dir = fromDir; + for (;;) { + const candidate = join(dir, "node_modules", name); + if (existsSync(candidate)) { + try { + return realpathSync(candidate); + } catch { + return candidate; + } + } + const parent = dirname(dir); + if (parent === dir) return null; + dir = parent; + } +} + +function probeFile(path) { + const candidates = [ + path, + `${path}.js`, + `${path}.mjs`, + `${path}.cjs`, + `${path}.json`, + `${path}.node`, + join(path, "index.js"), + join(path, "index.mjs"), + join(path, "index.cjs"), + join(path, "index.json"), + join(path, "index.node"), + ]; + for (const candidate of candidates) { + try { + const st = statSync(candidate); + if (st.isFile()) return candidate; + } catch { + // try next + } + } + return null; +} + +const EXPORT_CONDITIONS = new Set(["node", "import", "default", "require", "types"]); +const CONDITION_ORDER = ["node", "import", "require", "default"]; + +function resolveExports(exportsValue, subpath) { + if (typeof exportsValue === "string") { + return subpath === "." || subpath.startsWith("#") ? exportsValue : null; + } + if (Array.isArray(exportsValue)) { + for (const entry of exportsValue) { + const resolved = resolveExports(entry, subpath); + if (resolved) return resolved; + } + return null; + } + if (exportsValue === null || typeof exportsValue !== "object") return null; + // Path map vs condition object: keys starting with "." (or "#" for a + // package "imports" map) form a path map. + const keys = Object.keys(exportsValue); + const isPathMap = keys.some((key) => key === "." || key.startsWith("./") || key.startsWith("#")); + if (isPathMap) { + if (subpath in exportsValue) { + return resolveExports(exportsValue[subpath], "."); + } + for (const key of keys) { + const wildcard = key.indexOf("*"); + if (wildcard === -1 || !key.startsWith(".")) continue; + const prefix = key.slice(0, wildcard); + const suffix = key.slice(wildcard + 1); + if (subpath.startsWith(prefix) && subpath.endsWith(suffix) && subpath.length >= prefix.length + suffix.length) { + const replacement = subpath.slice(prefix.length, subpath.length - suffix.length); + const target = resolveExports(exportsValue[key], "."); + if (typeof target === "string") return target.replace("*", replacement); + } + } + return null; + } + for (const condition of CONDITION_ORDER) { + if (condition in exportsValue && EXPORT_CONDITIONS.has(condition)) { + const resolved = resolveExports(exportsValue[condition], subpath); + if (resolved) return resolved; + } + } + return null; +} + +/** + * Resolve the entry file for `subpath` inside the package at pkgDir. + * Returns { file } on success or { error } with a concrete reason. + * If the package has an `exports` map that does not expose the subpath, that + * is an error (matching node runtime behavior), not a silent fs fallback. + */ +function resolvePackageEntry(pkgDir, subpath) { + const pkgJsonPath = join(pkgDir, "package.json"); + if (!existsSync(pkgJsonPath)) return { error: `no package.json in ${pkgDir}` }; + let pkgJson; + try { + pkgJson = readJson(pkgJsonPath); + } catch (error) { + return { error: `unreadable package.json in ${pkgDir}: ${error.message}` }; + } + let target = null; + if (pkgJson.exports) { + target = resolveExports(pkgJson.exports, subpath); + if (!target && subpath !== ".") { + return { error: `exports map of ${pkgJson.name || pkgDir} does not expose subpath ${subpath}` }; + } + } + if (!target) { + if (subpath !== ".") { + target = `./${subpath.slice(2)}`; + } else { + target = pkgJson.main || pkgJson.module || "./index.js"; + } + } + if (typeof target !== "string") return { error: `unresolvable export target in ${pkgDir}` }; + const file = probeFile(resolve(pkgDir, target)); + if (!file) return { error: `entry ${target} of ${pkgJson.name || pkgDir} not found on disk` }; + return { file }; +} + +// --------------------------------------------------------------------------- +// Native artifact inspection +// --------------------------------------------------------------------------- + +const ELF_MACHINE_TO_ARCH = { 3: "ia32", 40: "arm", 62: "x64", 183: "arm64", 243: "riscv64" }; + +function inspectNativeBinary(file) { + let handle; + try { + handle = readFileSync(file); + } catch { + return { format: "unreadable" }; + } + if (handle.length >= 20 && handle[0] === 0x7f && handle[1] === 0x45 && handle[2] === 0x4c && handle[3] === 0x46) { + return { + format: "elf", + bits: handle[4] === 2 ? 64 : 32, + arch: ELF_MACHINE_TO_ARCH[handle.readUInt16LE(18)] || `unknown(${handle.readUInt16LE(18)})`, + }; + } + if (handle.length >= 4 && handle[0] === 0x00 && handle[1] === 0x61 && handle[2] === 0x73 && handle[3] === 0x6d) { + return { format: "wasm", arch: "wasm32 (portable)" }; + } + return { format: "unknown" }; +} + +/** Package root (bundle-relative, posix) of a file inside node_modules. */ +function packageRootOfRelative(relPath) { + const segments = relPath.split("/"); + const idx = segments.lastIndexOf("node_modules"); + if (idx === -1 || idx === segments.length - 1) return relPath; + const rest = segments.slice(idx + 1); + const root = rest[0].startsWith("@") && rest.length > 1 ? [rest[0], rest[1]] : [rest[0]]; + return [...segments.slice(0, idx + 1), ...root].join("/"); +} + +// --------------------------------------------------------------------------- +// Main packager +// --------------------------------------------------------------------------- + +function main() { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + console.log( + [ + "Usage: node package-sdk.mjs [options]", + " --source vendored pi monorepo (default: /vendor/pi)", + " --out bundle directory; must not exist or be empty (default: $PI_SDK_BUNDLE_OUT or /../tmp/pi-sdk--)", + " --target-arch arm64 (default) | x64", + " --target-platform linux (default; only supported value)", + " --skip-verify skip the post-build runtime verification", + " --no-tar do not create the .tar.gz transfer archive", + " --help show this help", + "", + "Nothing is ever deleted: preexisting outputs, archives, checksums and", + "sibling files are refused, and no pruning of older bundles happens.", + ].join("\n"), + ); + return; + } + + const hostArch = process.arch; + const source = realpathSync(options.source); + const outDir = options.out; + const startedAt = Date.now(); + + log("1/7", `source: ${source}`); + log("1/7", `out: ${outDir} (target ${options.targetPlatform}/${options.targetArch}, host ${process.platform}/${hostArch})`); + assertSafeOutput(source, outDir); + log("1/7", "output safety checks passed (empty/disjoint output, no symlink redirection, no preexisting targets)"); + + // --- Step 2: validate the built source tree -------------------------------- + + const monorepoJson = join(source, "package.json"); + if (!existsSync(monorepoJson) || readJson(monorepoJson).name !== "pi-monorepo") { + fail(`${source} is not the vendored pi monorepo root (expected package.json name "pi-monorepo")`); + } + const codingAgentDir = join(source, "packages", "coding-agent"); + for (const required of ["package.json", "dist", "dist/cli.js", "dist/index.js", "dist/rpc-entry.js"]) { + if (!existsSync(join(codingAgentDir, required))) { + fail( + `missing ${join(codingAgentDir, required)}: the vendored Pi SDK is not built. ` + + "Build it first (see CONTRIBUTING.md: vendor/pi npm ci + build:unbundled); this tool never builds or installs.", + ); + } + } + + // Workspace package index: realpath(packages/) -> package info. + const workspaceByDir = new Map(); + const workspaceByName = new Map(); + for (const entry of readdirSync(join(source, "packages"), { withFileTypes: true })) { + if (!entry.isDirectory()) continue; + const pkgDir = join(source, "packages", entry.name); + const pkgJsonPath = join(pkgDir, "package.json"); + if (!existsSync(pkgJsonPath)) continue; + let pkgJson; + try { + pkgJson = readJson(pkgJsonPath); + } catch { + continue; + } + if (!pkgJson.name) continue; + const info = { name: pkgJson.name, dir: realpathSync(pkgDir), pkgJson }; + workspaceByDir.set(info.dir, info); + workspaceByName.set(info.name, info); + } + + // --- Step 3: compute the runtime import closure ----------------------------- + + const includedWorkspaces = new Map(); // name -> info + const includedThirdParty = new Map(); // pkgRoot -> {name, version, license, sourceDir, bundleDir, reasons:[]} + const edges = []; + const unresolved = []; + const warnings = []; // classified warnings {from, spec, classification, reason} + const dynamicSiteRecords = []; // audited nonliteral sites {package, classification} + const scanQueue = []; + const scanned = new Set(); + + function bundleDirForThirdParty(pkgRoot) { + const underRoot = `${source}${sep}node_modules${sep}`; + if (pkgRoot.startsWith(underRoot)) { + return join(outDir, "node_modules", pkgRoot.slice(underRoot.length)); + } + for (const [, info] of workspaceByDir) { + const nested = `${info.dir}${sep}node_modules${sep}`; + if (pkgRoot.startsWith(nested)) { + return join(outDir, "node_modules", info.name, "node_modules", pkgRoot.slice(nested.length)); + } + } + return null; + } + + function enqueueWorkspaceDist(info) { + const distDir = join(info.dir, "dist"); + if (!existsSync(distDir)) { + fail(`workspace package ${info.name} is imported at runtime but has no dist/ build: ${info.dir}`); + } + const walk = (dir) => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const path = join(dir, entry.name); + if (entry.isDirectory()) { + // dist/bun only serves bun-compiled binaries; it is neither scanned nor shipped. + if (entry.name !== "bun") walk(path); + } else if (entry.name.endsWith(".js") && !entry.name.endsWith(".d.js")) scanQueue.push(path); + } + }; + walk(distDir); + } + + function includeWorkspace(info, reason) { + if (includedWorkspaces.has(info.name)) return; + includedWorkspaces.set(info.name, info); + log("3/7", `first-party: ${info.name} (${reason})`); + enqueueWorkspaceDist(info); + } + + function includeThirdParty(pkgRoot, importerLabel, spec) { + if (includedThirdParty.has(pkgRoot)) { + includedThirdParty.get(pkgRoot).reasons.push(`${importerLabel} -> ${spec}`); + return; + } + const pkgJsonPath = join(pkgRoot, "package.json"); + if (!existsSync(pkgRoot) || !existsSync(pkgJsonPath)) { + fail(`resolved ${spec} (from ${importerLabel}) to ${pkgRoot} which has no package.json`); + } + const bundleDir = bundleDirForThirdParty(pkgRoot); + if (!bundleDir) { + fail( + `resolved ${spec} (from ${importerLabel}) to ${pkgRoot}, which is neither under the monorepo node_modules ` + + "nor under a workspace package's nested node_modules; refusing to bundle unknown layouts", + ); + } + const pkgJson = readJson(pkgJsonPath); + includedThirdParty.set(pkgRoot, { + name: pkgJson.name || basename(pkgRoot), + version: pkgJson.version || "0.0.0", + license: pkgJson.license || "UNKNOWN", + sourceDir: pkgRoot, + bundleDir, + reasons: [`${importerLabel} -> ${spec}`], + }); + log("3/7", `third-party: ${pkgJson.name || basename(pkgRoot)}@${pkgJson.version || "?"} (${importerLabel} -> ${spec})`); + // Scan every shipped code file of the package so the closure covers + // lazily required internals, not just the entry chain. + const walk = (dir, depth) => { + let entries; + try { + entries = readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const entry of entries) { + const path = join(dir, entry.name); + if (entry.isDirectory()) { + if (thirdPartyKeep(entry.name, true, depth)) walk(path, depth + 1); + } else if (entry.isFile() && thirdPartyKeep(entry.name, false, depth)) { + if (/\.(js|mjs|cjs)$/.test(entry.name)) scanQueue.push(path); + } + } + }; + walk(pkgRoot, 0); + } + + function classifyResolvedFile(file, importerLabel, spec) { + const realFile = realpathSync(file); + // Workspace package? + let dir = dirname(realFile); + for (;;) { + if (workspaceByDir.has(dir)) { + const info = workspaceByDir.get(dir); + edges.push({ from: importerLabel, to: info.name, spec }); + includeWorkspace(info, `${importerLabel} -> ${spec}`); + return; + } + if (dir === dirname(dir)) break; + dir = dirname(dir); + } + // Third-party package: ascend to the package root under a node_modules directory + // (scoped packages occupy two path segments: @scope/name). + dir = dirname(realFile); + for (;;) { + const parent = dirname(dir); + if (basename(parent) === "node_modules") { + const segments = relative(parent, realFile).split(sep); + const packageRoot = segments[0].startsWith("@") ? join(parent, segments[0], segments[1]) : join(parent, segments[0]); + const pkgJsonPath = join(packageRoot, "package.json"); + const name = existsSync(pkgJsonPath) ? readJson(pkgJsonPath).name || basename(packageRoot) : basename(packageRoot); + edges.push({ from: importerLabel, to: name, spec }); + includeThirdParty(packageRoot, importerLabel, spec); + return; + } + if (parent === dir) break; + dir = parent; + } + // File inside a workspace package tree (relative import within dist). + dir = dirname(realFile); + for (;;) { + if (workspaceByDir.has(dir)) { + const info = workspaceByDir.get(dir); + if (!includedWorkspaces.has(info.name)) { + fail(`file ${realFile} lies inside workspace package ${info.name} which is not part of the closure`); + } + return; + } + if (dir === dirname(dir)) break; + dir = dirname(dir); + } + unresolved.push(`${importerLabel}: cannot classify import ${spec} -> ${realFile} (file outside every included package)`); + } + + function owningPackageInfo(file) { + // Prefer workspace info, then third-party package root by node_modules ascent. + let dir = dirname(realpathSync(file)); + for (;;) { + if (workspaceByDir.has(dir)) { + const info = workspaceByDir.get(dir); + return { name: info.name, root: info.dir, pkgJson: info.pkgJson, firstParty: true }; + } + const parent = dirname(dir); + if (basename(parent) === "node_modules") { + const segments = relative(parent, dir).split(sep); + const root = segments[0].startsWith("@") && segments.length > 1 ? join(parent, segments[0], segments[1]) : dir; + const pkgJsonPath = join(root, "package.json"); + const pkgJson = existsSync(pkgJsonPath) ? readJson(pkgJsonPath) : null; + return { name: pkgJson?.name || basename(root), root, pkgJson, firstParty: false }; + } + if (parent === dir) return null; + dir = parent; + } + } + + function classifyUnresolvedImport(file, importerLabel, spec, detail) { + const owner = owningPackageInfo(file); + const specRoot = spec.startsWith(".") || spec.startsWith("#") ? spec : spec.split("/").slice(0, spec.startsWith("@") ? 2 : 1).join("/"); + let entry = OPTIONAL_IMPORTS[specRoot]; + if (!entry && owner && !owner.firstParty) { + entry = OPTIONAL_IMPORTS.byImporter[owner.name]?.[spec]; + } + if (entry) { + warnings.push({ from: importerLabel, spec, classification: entry.classification, reason: entry.reason }); + return; + } + unresolved.push( + `${importerLabel}: ${detail}` + + (owner ? ` (owning package: ${owner.name}${owner.firstParty ? " [first-party]" : ""})` : "") + + `; unclassified: add an audited classification or fix the import`, + ); + } + + const codingAgentInfo = workspaceByName.get("@earendil-works/pi-coding-agent"); + if (!codingAgentInfo) fail("packages/coding-agent not found in the vendored source"); + includeWorkspace(codingAgentInfo, "bundle root (SDK + CLI)"); + + while (scanQueue.length > 0) { + const file = scanQueue.pop(); + if (scanned.has(file)) continue; + scanned.add(file); + const importerLabel = relative(source, file) || file; + const { specifiers, dynamicSites } = scanFile(file); + for (const spec of specifiers) { + if (isBuiltinSpecifier(spec)) continue; + if (spec.startsWith(".")) { + const resolved = probeFile(resolve(dirname(file), spec)); + if (!resolved) { + // A missing relative import is never silently ignored. + const detail = `relative import ${spec} does not resolve to a file`; + const owner = owningPackageInfo(file); + if (owner && !owner.firstParty) classifyUnresolvedImport(file, importerLabel, spec, detail); + else unresolved.push(`${importerLabel}: ${detail} (first-party; the bundle would be broken)`); + continue; + } + scanQueue.push(resolved); + continue; + } + if (spec.startsWith("#")) { + // Package-internal "imports" map (e.g. chalk's #ansi-styles). + const owner = owningPackageInfo(file); + const target = owner?.pkgJson?.imports ? resolveExports(owner.pkgJson.imports, spec) : null; + const resolvedFile = owner && target ? probeFile(resolve(owner.root, target)) : null; + if (!resolvedFile) { + const detail = `cannot resolve imports-map entry ${spec} of ${owner?.root ?? "unknown package"}`; + if (owner && !owner.firstParty) classifyUnresolvedImport(file, importerLabel, spec, detail); + else unresolved.push(`${importerLabel}: ${detail} (first-party)`); + continue; + } + classifyResolvedFile(resolvedFile, importerLabel, spec); + continue; + } + // Bare specifier: package name + optional subpath. + const parts = spec.split("/"); + const pkgName = spec.startsWith("@") ? parts.slice(0, 2).join("/") : parts[0]; + const subpath = "." + spec.slice(pkgName.length); + const pkgDir = findPackageDir(dirname(file), pkgName); + if (!pkgDir) { + classifyUnresolvedImport(file, importerLabel, spec, `package not found in any node_modules: ${spec}`); + continue; + } + const entryResult = resolvePackageEntry(pkgDir, subpath); + if (!entryResult.file) { + classifyUnresolvedImport(file, importerLabel, spec, `${entryResult.error} (specifier ${spec})`); + continue; + } + classifyResolvedFile(entryResult.file, importerLabel, spec); + } + // Nonliteral dynamic import/require sites must be audited per package. + if (dynamicSites.length > 0) { + const owner = owningPackageInfo(file); + const audit = owner ? DYNAMIC_IMPORT_AUDIT[owner.name] : null; + if (!audit) { + unresolved.push( + `${importerLabel}: unaudited nonliteral dynamic import site(s): ${dynamicSites.map((s) => `require/import(${s})`).join(", ")}` + + (owner ? ` (owning package: ${owner.name})` : "") + + "; computed import roots must be audited in DYNAMIC_IMPORT_AUDIT", + ); + } else { + dynamicSiteRecords.push({ + from: importerLabel, + package: owner.name, + sites: dynamicSites.length, + classification: audit.classification, + reason: audit.reason, + }); + } + } + } + + if (unresolved.length > 0) { + console.error("package-sdk: unexplained missing runtime edges (build stopped):"); + for (const line of unresolved) console.error(` ${line}`); + fail(`${unresolved.length} unexplained import edge(s); bundle would be incomplete`); + } + if (warnings.length > 0) { + log("3/7", `${warnings.length} classified optional/unresolvable import(s) recorded in the manifest`); + } + log( + "3/7", + `closure: ${includedWorkspaces.size} first-party, ${includedThirdParty.size} third-party packages, ${scanned.size} files scanned, ${dynamicSiteRecords.length} audited dynamic-site file(s)`, + ); + + // --- Step 4: copy into the bundle ------------------------------------------- + + // assertSafeOutput proved outDir absent or empty; create it (parents included). + mkdirSync(outDir, { recursive: true }); + writeFileSync( + join(outDir, BUNDLE_MARKER), + `${JSON.stringify({ generator: "portable/raspberry-pi/package-sdk.mjs", targetPlatform: options.targetPlatform, targetArch: options.targetArch }, null, "\t")}\n`, + ); + + function copyFileSyncChecked(from, to) { + mkdirSync(dirname(to), { recursive: true }); + const fromStat = lstatSync(from); + if (fromStat.isSymbolicLink()) { + fail(`symlink in copied source tree (bundle format forbids symlinks): ${from} -> ${readlinkSync(from)}`); + } + if (!fromStat.isFile()) { + fail(`non-regular file in copied source tree: ${from}`); + } + copyFileSync(from, to); + chmodSync(to, fromStat.mode & 0o777); + } + + function copyTree(fromDir, toDir, keep, depth = 0) { + for (const entry of readdirSync(fromDir, { withFileTypes: true })) { + const from = join(fromDir, entry.name); + const to = join(toDir, entry.name); + if (entry.isSymbolicLink()) { + // Simplest portable format: no symlinks at all. + fail(`symlink in copied source tree (bundle format forbids symlinks): ${from} -> ${readlinkSync(from)}`); + } + if (entry.isDirectory()) { + if (keep(entry.name, true, depth)) copyTree(from, to, keep, depth + 1); + continue; + } + if (entry.isFile()) { + if (keep(entry.name, false, depth)) copyFileSyncChecked(from, to); + continue; + } + fail(`non-regular file in copied source tree: ${from}`); + } + } + + const workspaceKeep = (name, isDir) => { + if (!isDir) return !name.endsWith(".map"); + return name !== "bun"; // dist/bun only serves bun-compiled binaries + }; + + function copyWorkspacePackage(info) { + const dest = join(outDir, "node_modules", info.name); + for (const metaFile of ["package.json", "README.md", "CHANGELOG.md"]) { + const from = join(info.dir, metaFile); + if (existsSync(from)) copyFileSyncChecked(from, join(dest, metaFile)); + } + copyTree(join(info.dir, "dist"), join(dest, "dist"), workspaceKeep); + // Runtime native prebuilds (pi-tui clipboard helper). The target-arch + // prebuild is REQUIRED; the host-arch prebuild (when different) is + // included so host verification can actually load a binding. + if (info.name === "@earendil-works/pi-tui") { + const hostPrebuildLimitations = []; + for (const arch of [options.targetArch, ...(hostArch !== options.targetArch ? [hostArch] : [])]) { + const prebuildDir = join(info.dir, "native", options.targetPlatform, "prebuilds", `${options.targetPlatform}-${arch}`); + if (!existsSync(prebuildDir)) { + if (arch === options.targetArch) { + fail( + `pi-tui prebuild for ${options.targetPlatform}-${arch} is missing at ${prebuildDir}; ` + + "no target-arch native clipboard helper can be bundled (concrete blocker)", + ); + } + hostPrebuildLimitations.push( + `host-arch prebuild ${options.targetPlatform}-${hostArch} absent; host-side native load check will be skipped`, + ); + continue; + } + copyTree(prebuildDir, join(dest, "native", options.targetPlatform, "prebuilds", `${options.targetPlatform}-${arch}`), () => true); + } + for (const limitation of hostPrebuildLimitations) warnings.push({ from: "package-sdk", spec: "pi-tui prebuilds", classification: "host-prebuild-absent", reason: limitation }); + } + // Third-party notices shipped by first-party packages. + if (info.name === "@earendil-works/pi-mcp") { + const licensesDir = join(info.dir, "LICENSES"); + if (existsSync(licensesDir)) copyTree(licensesDir, join(dest, "LICENSES"), () => true); + } + // Runtime-referenced asset directories of the coding agent (docs and + // examples commands read these; images are documentation-only). + if (info.name === "@earendil-works/pi-coding-agent") { + if (existsSync(join(info.dir, "docs"))) { + copyTree(join(info.dir, "docs"), join(dest, "docs"), (name, isDir) => !(isDir && name === "images")); + } + if (existsSync(join(info.dir, "examples"))) { + copyTree(join(info.dir, "examples"), join(dest, "examples"), (name, isDir) => + isDir ? name !== "node_modules" : !name.endsWith(".map"), + ); + } + } + } + + for (const [, info] of includedWorkspaces) copyWorkspacePackage(info); + for (const [, info] of includedThirdParty) { + copyTree(info.sourceDir, info.bundleDir, thirdPartyKeep); + } + + // First-party license + verification tool. + copyFileSyncChecked(join(source, "LICENSE"), join(outDir, "LICENSE")); + copyFileSyncChecked(resolve(SCRIPT_DIR, "verify-bundle.mjs"), join(outDir, "verify", "verify-bundle.mjs")); + + // --- Step 5: read-back verification, native inventory, manifest ------------- + + log("5/7", "inventorying bundle files"); + const problems = []; + + function inventoryBundle() { + const fileEntries = []; + const nativeArtifacts = []; + const walk = (dir) => { + for (const entry of readdirSync(dir, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) { + const path = join(dir, entry.name); + const rel = relative(outDir, path); + if (entry.isDirectory()) { + walk(path); + continue; + } + if (entry.isSymbolicLink()) { + problems.push(`symlink in bundle (format forbids symlinks): ${rel} -> ${readlinkSync(path)}`); + continue; + } + if (!entry.isFile()) { + problems.push(`unexpected file type in bundle: ${rel}`); + continue; + } + const data = readFileSync(path); + const sha256 = createHash("sha256").update(data).digest("hex"); + fileEntries.push({ path: rel, size: data.length, sha256 }); + if (rel.endsWith(".node") || rel.endsWith(".wasm") || rel.endsWith(".so")) { + const info = inspectNativeBinary(path); + nativeArtifacts.push({ path: rel, ...info }); + } + } + }; + walk(outDir); + return { fileEntries, nativeArtifacts }; + } + + const { fileEntries, nativeArtifacts } = inventoryBundle(); + + // Native coverage policy: every ELF must be target-arch, or host-arch with + // a target-arch sibling in the same package. Wrong-arch-only is a blocker. + const elfByPackage = new Map(); + for (const artifact of nativeArtifacts) { + if (artifact.format !== "elf") continue; + const root = packageRootOfRelative(artifact.path); + if (!elfByPackage.has(root)) elfByPackage.set(root, new Set()); + elfByPackage.get(root).add(artifact.arch); + } + for (const artifact of nativeArtifacts) { + if (artifact.format === "wasm") continue; + if (artifact.format !== "elf") { + problems.push(`native artifact ${artifact.path} has unrecognized format ${artifact.format}`); + continue; + } + if (artifact.arch === options.targetArch) { + artifact.coverage = "target"; + continue; + } + if (artifact.arch === hostArch) { + const root = packageRootOfRelative(artifact.path); + const archs = elfByPackage.get(root); + if (archs.has(options.targetArch)) { + artifact.coverage = `host (${options.targetArch} sibling present in same package)`; + continue; + } + problems.push( + `native artifact ${artifact.path} is ${artifact.arch}-only: package ${root} ships no ${options.targetArch} ` + + `variant. This bundle could not run on the ${options.targetArch} target. Concrete blocker: obtain a ` + + `${options.targetArch} build of this dependency and rebuild the bundle.`, + ); + continue; + } + problems.push(`native artifact ${artifact.path} is ${artifact.arch}; neither target (${options.targetArch}) nor host (${hostArch})`); + } + + if (problems.length > 0) { + console.error("package-sdk: bundle inventory problems:"); + for (const problem of problems) console.error(` ${problem}`); + fail(`${problems.length} bundle problem(s)`); + } + + const totals = fileEntries.reduce((acc, entry) => ({ files: acc.files + 1, bytes: acc.bytes + entry.size }), { + files: 0, + bytes: 0, + }); + + // Provenance: the checkout may not be a git repo; digests are the record. + const gitRev = (() => { + const result = spawnSync("git", ["-C", source, "rev-parse", "HEAD"], { encoding: "utf8" }); + return result.status === 0 ? result.stdout.trim() : null; + })(); + const provenance = { + sourceRoot: source, + repositoryRevision: gitRev, + repositoryRevisionNote: gitRev ? "git rev-parse of the vendored source" : "not a git checkout; the digests below are the source provenance", + sourceDigests: { + "package.json": sha256File(join(source, "package.json")), + "package-lock.json": existsSync(join(source, "package-lock.json")) ? sha256File(join(source, "package-lock.json")) : null, + "packages/coding-agent/package.json": sha256File(join(codingAgentDir, "package.json")), + }, + firstPartyVersions: Object.fromEntries( + [...includedWorkspaces.values()].sort((a, b) => a.name.localeCompare(b.name)).map((info) => [info.name, info.pkgJson.version || "?"]), + ), + generator: { + script: "portable/raspberry-pi/package-sdk.mjs", + sha256: sha256File(fileURLToPath(import.meta.url)), + verifierSha256: sha256File(resolve(SCRIPT_DIR, "verify-bundle.mjs")), + }, + buildHost: { platform: process.platform, arch: process.arch, nodeVersion: process.version }, + generatedAt: new Date().toISOString(), + }; + + const manifest = { + generator: "portable/raspberry-pi/package-sdk.mjs", + created: new Date().toISOString(), + status: "content-complete (integrity root is manifest.txt; rerun verify/verify-bundle.mjs on the target)", + scope: "Static import closure of @earendil-works/pi-coding-agent plus runtime verification of the SDK import, quickjs instantiation and the write tool. NOT a claim of universal SDK completeness; unexercised paths may still miss runtime edges.", + integrityNote: + "manifest.txt is the integrity root of the bundle contents. It is integrity relative to the separately trusted transfer digest (the .tar.gz.sha256 sidecar verified over a trusted channel); it is not authenticity.", + target: { + platform: options.targetPlatform, + arch: options.targetArch, + nodeEngine: ">=22.19.0 (per @earendil-works/pi-coding-agent engines)", + nativePolicy: `every ELF is ${options.targetArch}, or ${hostArch} with a ${options.targetArch} sibling in the same package; host verification loads only ${hostArch} bindings, ${options.targetArch} binding load is an open on-target gate`, + }, + provenance, + packages: { + firstParty: [...includedWorkspaces.keys()].sort().map((name) => { + const info = includedWorkspaces.get(name); + return { name, version: info.pkgJson.version, license: info.pkgJson.license || "MIT", reason: "runtime import closure from @earendil-works/pi-coding-agent" }; + }), + thirdParty: [...includedThirdParty.values()] + .sort((a, b) => a.name.localeCompare(b.name)) + .map((info) => ({ + name: info.name, + version: info.version, + license: info.license, + bundleDir: relative(outDir, info.bundleDir), + reasons: [...new Set(info.reasons)].slice(0, 4), + })), + }, + nativeArtifacts, + importEdges: [...new Set(edges.map((edge) => `${edge.from} -> ${edge.to} via ${edge.spec}`))].sort(), + warnings, + auditedDynamicImports: dynamicSiteRecords, + totals, + }; + writeFileSync(join(outDir, "manifest.json"), `${JSON.stringify(manifest, null, "\t")}\n`); + + // Third-party notices. License fields are package.json declarations, not + // verified notice text; shipped license files are preserved in place. + const notices = [ + "# Third-party notices", + "", + "This bundle redistributes the third-party npm packages listed below as part of", + "the Pi SDK runtime closure. The License column is the package.json declaration", + "(metadata, not notice provenance); license/notice files shipped inside the", + "packages are preserved in place within node_modules/.", + "", + "| Package | Version | License (declared) | Bundle path |", + "| --- | --- | --- | --- |", + ...[...includedThirdParty.values()] + .sort((a, b) => a.name.localeCompare(b.name)) + .map((info) => `| ${info.name} | ${info.version} | ${info.license} | ${relative(outDir, info.bundleDir)} |`), + "", + "First-party code (@earendil-works/*, LICENSE in the bundle root) is MIT licensed.", + "", + ]; + writeFileSync(join(outDir, "THIRD_PARTY_NOTICES.md"), notices.join("\n")); + + // manifest.txt: the integrity root. Hash EVERY file except manifest.txt + // itself (written last, after all content exists). + const { fileEntries: finalEntries } = inventoryBundle(); + const manifestTxt = [ + `# Pi SDK runtime bundle manifest (${options.targetPlatform}/${options.targetArch})`, + "# sha256 of every file in this bundle except this file itself.", + "# Verify with: sha256sum -c manifest.txt (from the bundle root)", + "# Integrity relative to the trusted .tar.gz.sha256 transfer digest; not authenticity.", + ...finalEntries.filter((entry) => entry.path !== "manifest.txt").map((entry) => `${entry.sha256} ${entry.path}`), + "", + ].join("\n"); + writeFileSync(join(outDir, "manifest.txt"), manifestTxt); + + log("5/7", `bundle: ${totals.files} files, ${(totals.bytes / 1024 / 1024).toFixed(1)} MiB, ${nativeArtifacts.length} native/wasm artifacts`); + + // --- Step 6: runtime verification on this host ------------------------------- + + let verification = { skipped: true, note: "host verification only; never proof for the target architecture" }; + if (!options.skipVerify) { + log("6/7", `running bundle verification on this host (${hostArch}; target ${options.targetArch} stays an open gate)`); + const verifyScript = join(outDir, "verify", "verify-bundle.mjs"); + const result = spawnSync(process.execPath, [verifyScript, "--bundle", outDir, "--check-manifest"], { + encoding: "utf8", + timeout: 180000, + // The verifier re-sanitizes its own environment; start it clean anyway. + env: { + PATH: process.env.PATH, + HOME: process.env.HOME, + TMPDIR: process.env.TMPDIR, + LANG: process.env.LANG, + TZ: process.env.TZ, + }, + }); + process.stdout.write(result.stdout || ""); + if (result.status !== 0) { + process.stderr.write(result.stderr || ""); + fail(`bundle verification failed with exit code ${result.status}; see output above (bundle left in place for inspection)`); + } + verification = { skipped: false, exitCode: result.status, host: `${process.platform}/${hostArch}`, target: `${options.targetPlatform}/${options.targetArch}` }; + } else { + log("6/7", "verification skipped (--skip-verify)"); + } + + // --- Step 7: transfer archive + external summary (no pruning, ever) ---------- + + let archive = null; + if (!options.noTar) { + const tarPath = `${outDir}.tar.gz`; + log("7/7", `creating ${tarPath}`); + const tarBase = basename(outDir); + // Deterministic flags only; NO silent fallback that would change the + // layout. If this tar does not support them, fail explicitly. + const tarResult = spawnSync( + "tar", + [ + "--sort=name", + "--owner=0", + "--group=0", + "--numeric-owner", + "--mtime=1970-01-01T00:00:00Z", + "-czf", + tarPath, + "-C", + dirname(outDir), + tarBase, + ], + { encoding: "utf8" }, + ); + if (tarResult.status !== 0) { + process.stderr.write(tarResult.stderr || ""); + fail("tar failed (deterministic flags not supported?); no fallback archive is created. Bundle left in place."); + } + const sha = createHash("sha256").update(readFileSync(tarPath)).digest("hex"); + writeFileSync(`${tarPath}.sha256`, `${sha} ${basename(tarPath)}\n`); + archive = { path: tarPath, bytes: statSync(tarPath).size, sha256: sha }; + log("7/7", `archive sha256 (transfer digest): ${sha}`); + // The build summary lives NEXT TO the archive, never inside the bundle: + // the archive must contain exactly the hashed content. + const summary = { + bundleDir: outDir, + archive, + totals, + firstPartyPackages: [...includedWorkspaces.keys()].sort(), + thirdPartyPackages: [...includedThirdParty.values()].map((info) => `${info.name}@${info.version}`).sort(), + nativeArtifacts, + verification, + openGates: [ + `target ${options.targetArch} native bindings are not loadable on this ${hostArch} host; run verify/verify-bundle.mjs on the target`, + ], + elapsedMs: Date.now() - startedAt, + }; + writeFileSync(`${outDir}.build-summary.json`, `${JSON.stringify(summary, null, "\t")}\n`); + } else { + log("7/7", "archive skipped (--no-tar)"); + } + + log("done", `${outDir} (${(totals.bytes / 1024 / 1024).toFixed(1)} MiB, ${((Date.now() - startedAt) / 1000).toFixed(1)}s); nothing was pruned`); +} + +try { + main(); +} catch (error) { + if (error instanceof PackError) { + console.error(`package-sdk: error: ${error.message}`); + process.exit(1); + } + throw error; +} diff --git a/portable/raspberry-pi/provision-guest.sh b/portable/raspberry-pi/provision-guest.sh new file mode 100755 index 0000000..9005164 --- /dev/null +++ b/portable/raspberry-pi/provision-guest.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Run inside the disposable ARM64 guest as the rehearsal user. +set -euo pipefail +[[ $(uname -m) == aarch64 ]] || { echo 'ARM64 guest required' >&2; exit 1; } +node_version=v24.22.0 +node_archive="node-$node_version-linux-arm64.tar.xz" +node_sha=fef00bb4d36e1c50104a9775a10359beee98f162b4104dd12472c75065c6a143 +mkdir -p "$HOME/.local/opt" "$HOME/workspace" "$HOME/provision-cache" +cd "$HOME/provision-cache" +curl --fail --location --retry 3 "https://nodejs.org/dist/$node_version/$node_archive" -o "$node_archive" +printf '%s %s\n' "$node_sha" "$node_archive" | sha256sum --check - +tar -xJf "$node_archive" -C "$HOME/.local/opt" +[[ -f "$HOME/portable/workspace/server.mjs" ]] || { echo 'Deploy portable/workspace first' >&2; exit 1; } +sudo tee /etc/systemd/system/portable-workspace.service >/dev/null < "$HOME/.xinitrc" <<'XINIT' +#!/bin/sh +xset -dpms +xset s off +openbox & +exec chromium --kiosk --no-first-run --disable-session-crashed-bubble --user-data-dir=/tmp/portable-chromium http://127.0.0.1:8765 +XINIT + chmod +x "$HOME/.xinitrc" + mkdir -p "$HOME/.config" + cat > "$HOME/.bash_profile" <<'PROFILE' +if [ "$(tty)" = /dev/tty1 ]; then + exec startx -- -nolisten tcp +fi +PROFILE + sudo mkdir -p /etc/systemd/system/getty@tty1.service.d + sudo tee /etc/systemd/system/getty@tty1.service.d/autologin.conf >/dev/null < "$root/evidence/tool-packages.sha256" + ldd "$qemu" > "$root/evidence/qemu-ldd.txt" + if grep -q 'not found' "$root/evidence/qemu-ldd.txt"; then cat "$root/evidence/qemu-ldd.txt"; exit 1; fi + "$qemu" --version | tee "$root/evidence/qemu-version.txt" + ;; + image) + cd "$root/images" + if [[ ! -f "$image" ]]; then + curl --fail --location --retry 3 "$image_url" -o "$image.part" + printf '%s %s\n' "$image_sha" "$image.part" | sha512sum --check - + mv "$image.part" "$image" + fi + printf '%s %s\n' "$image_sha" "$image" | sha512sum --check - + printf '%s\n%s %s\n' "$image_url" "$image_sha" "$image" > "$root/evidence/image-provenance.txt" + ;; + prepare) + [[ ! -e "$root/images/run.qcow2" ]] || { echo 'Existing run.qcow2: retain it for recovery or remove explicitly before a fresh run.' >&2; exit 1; } + printf '%s %s\n' "$image_sha" "$root/images/$image" | sha512sum --check - + mkdir -p "$root/seed" + ssh-keygen -q -t ed25519 -N '' -f "$root/seed/guest-key" + printf 'instance-id: portable-arm64-rehearsal\nlocal-hostname: portable-rehearsal\n' > "$root/seed/meta-data" + cat > "$root/seed/user-data" < /dev/ttyAMA0'] +YAML + genisoimage -quiet -output "$root/images/seed.iso" -volid cidata -joliet -rock "$root/seed/user-data" "$root/seed/meta-data" + qemu-img create -f qcow2 -F qcow2 -b "$root/images/$image" "$root/images/run.qcow2" 12G + cp "$firmware/AAVMF_VARS.fd" "$root/images/vars.fd" + ;; + run) + exec "$qemu" -name portable-arm64-rehearsal \ + -machine virt -cpu cortex-a72 -accel tcg,thread=multi -smp 4 -m 4096 \ + -drive if=pflash,format=raw,readonly=on,file="$firmware/AAVMF_CODE.fd" \ + -drive if=pflash,format=raw,file="$root/images/vars.fd" \ + -drive if=none,id=os,format=qcow2,file="$root/images/run.qcow2" \ + -device virtio-blk-pci,drive=os \ + -drive if=none,id=seed,format=raw,readonly=on,file="$root/images/seed.iso" \ + -device virtio-blk-pci,drive=seed \ + -netdev user,id=net0,hostfwd=tcp:127.0.0.1:2222-:22 \ + -device virtio-net-pci,netdev=net0 \ + -device virtio-gpu-pci -device qemu-xhci -device usb-kbd -device usb-tablet \ + -vnc 127.0.0.1:1 -serial file:"$root/evidence/serial.log" \ + -qmp unix:"$root/tmp/qmp.sock",server=on,wait=off \ + -pidfile "$root/tmp/qemu.pid" -monitor none + ;; + ssh) + shift + exec ssh -i "$root/seed/guest-key" -o IdentitiesOnly=yes \ + -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile="$root/seed/known_hosts" \ + -p 2222 rehearsal@127.0.0.1 "$@" + ;; + *) echo "Usage: PORTABLE_REHEARSAL_DIR=/dedicated/storage $0 {tools|image|prepare|run|ssh [command]}"; exit 2 ;; +esac diff --git a/portable/raspberry-pi/sdk-package-tests.mjs b/portable/raspberry-pi/sdk-package-tests.mjs new file mode 100644 index 0000000..b0480de --- /dev/null +++ b/portable/raspberry-pi/sdk-package-tests.mjs @@ -0,0 +1,503 @@ +#!/usr/bin/env node +/** + * Synthetic-tree tests for portable/raspberry-pi/package-sdk.mjs and + * verify-bundle.mjs. + * + * All trees are disposable and created under a single mkdtemp sandbox; no user + * data is touched. Two groups: + * + * A. Packager output safety + closure policy (synthetic monorepo trees): + * sentinel/sibling survival (no pruning, ever), nonempty/inside/ancestor/ + * symlink/symlinked-ancestor/preexisting-target refusal, missing relative + * runtime edge, nonliteral dynamic import audit (audited passes, + * unaudited fails), classified optional import (classified passes, + * unclassified fails), wrong-arch-only native refusal (target-only + * passes). + * + * B. verify-bundle manifest rejection BEFORE any side-effect import: the + * fake SDK entry writes a marker file at import time; every rejection + * case must exit nonzero WITHOUT the marker (import never ran), and a + * clean manifest must produce the marker (import did run, order proven). + * + * Usage: node sdk-package-tests.mjs + */ +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; + +const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url)); +const PACKAGER = join(SCRIPT_DIR, "package-sdk.mjs"); +const VERIFIER = join(SCRIPT_DIR, "verify-bundle.mjs"); + +const results = []; +function record(name, passed, detail = "") { + results.push({ name, passed, detail }); + console.log(`${passed ? "PASS" : "FAIL"} ${name}${detail ? ` — ${detail}` : ""}`); +} + +function runTool(tool, args, extraEnv = {}) { + const result = spawnSync(process.execPath, [tool, ...args], { + encoding: "utf8", + timeout: 120000, + env: { ...process.env, ...extraEnv }, + }); + return { status: result.status, stdout: result.stdout || "", stderr: result.stderr || "" }; +} + +function expectFail(name, result, needle, sandbox) { + const out = `${result.stdout}${result.stderr}`; + const passed = result.status !== 0 && out.includes(needle); + record(name, passed, passed ? `refused: "${needle.split("\n")[0].slice(0, 90)}"` : `status=${result.status}, output did not contain ${JSON.stringify(needle.slice(0, 60))}: ${out.slice(0, 200).replace(/\n/g, " | ")}`); + if (sandbox) rmSync(sandbox, { recursive: true, force: true }); +} + +function expectPass(name, result, sandbox) { + const passed = result.status === 0; + record(name, passed, passed ? "" : `status=${result.status}: ${(result.stderr || result.stdout).slice(0, 300).replace(/\n/g, " | ")}`); + if (sandbox) rmSync(sandbox, { recursive: true, force: true }); +} + +// --------------------------------------------------------------------------- +// Synthetic source tree helpers +// --------------------------------------------------------------------------- + +function writeJson(file, value) { + writeFileSync(file, `${JSON.stringify(value, null, "\t")}\n`); +} + +function makeSyntheticSource(root, { distIndex = 'export const VERSION = "1.1.0-test";\n' } = {}) { + const src = join(root, "src"); + mkdirSync(join(src, "packages", "coding-agent", "dist"), { recursive: true }); + writeJson(join(src, "package.json"), { name: "pi-monorepo", version: "0.0.0" }); + writeJson(join(src, "packages", "coding-agent", "package.json"), { + name: "@earendil-works/pi-coding-agent", + version: "1.1.0", + license: "MIT", + }); + writeFileSync(join(src, "packages", "coding-agent", "dist", "index.js"), distIndex); + writeFileSync(join(src, "packages", "coding-agent", "dist", "cli.js"), "export {};\n"); + writeFileSync(join(src, "packages", "coding-agent", "dist", "rpc-entry.js"), "export {};\n"); + writeFileSync(join(src, "LICENSE"), "MIT test license\n"); + return src; +} + +function addThirdParty(src, name, mainJs, extraFiles = {}) { + const pkgDir = join(src, "node_modules", ...name.split("/")); + mkdirSync(pkgDir, { recursive: true }); + writeJson(join(pkgDir, "package.json"), { name, version: "1.0.0", license: "MIT", main: "index.js" }); + writeFileSync(join(pkgDir, "index.js"), mainJs); + for (const [rel, content] of Object.entries(extraFiles)) { + const file = join(pkgDir, rel); + mkdirSync(dirname(file), { recursive: true }); + content.buffer ? writeFileSync(file, content.buffer) : writeFileSync(file, content); + } + return pkgDir; +} + +/** Minimal ELF header (20 bytes) with the given machine architecture. */ +function elfHeader(machine) { + const buf = Buffer.alloc(20); + buf[0] = 0x7f; + buf.write("ELF", 1, "ascii"); + buf[4] = 2; // 64-bit + buf.writeUInt16LE(machine, 18); + return buf; +} +const EM_X64 = 62; +const EM_ARM64 = 183; + +// --------------------------------------------------------------------------- +// Group A: packager safety + closure policy +// --------------------------------------------------------------------------- + +function testPackagerSafety() { + // A1: sentinel + sibling archive survival; no pruning. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-a1-")); + try { + const src = makeSyntheticSource(sandbox); + const outParent = join(sandbox, "out"); + mkdirSync(join(outParent, "sentinel-dir"), { recursive: true }); + writeFileSync(join(outParent, "sentinel-dir", "keep.txt"), "keep me\n"); + writeFileSync(join(outParent, "pi-sdk-old.tar.gz"), "old archive bytes\n"); + mkdirSync(join(outParent, "pi-sdk-old"), { recursive: true }); + writeFileSync(join(outParent, "pi-sdk-old", ".pi-sdk-bundle"), "{}\n"); // marker-bearing old bundle + writeFileSync(join(outParent, "pi-sdk-old", "payload.txt"), "old payload\n"); + const result = runTool(PACKAGER, ["--source", src, "--out", join(outParent, "new-bundle"), "--skip-verify", "--no-tar"]); + const kept = + existsSync(join(outParent, "sentinel-dir", "keep.txt")) && + existsSync(join(outParent, "pi-sdk-old.tar.gz")) && + existsSync(join(outParent, "pi-sdk-old", "payload.txt")); + const built = existsSync(join(outParent, "new-bundle", "manifest.txt")); + record("A1 sentinel/sibling survival (no pruning)", result.status === 0 && kept && built, result.status === 0 && kept && built ? "" : `status=${result.status} kept=${kept} built=${built}: ${result.stderr.slice(0, 200)}`); + } finally { + rmSync(sandbox, { recursive: true, force: true }); + } + } + + // A2: refuse nonempty output. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-a2-")); + const src = makeSyntheticSource(sandbox); + const out = join(sandbox, "nonempty"); + mkdirSync(out, { recursive: true }); + writeFileSync(join(out, "precious.txt"), "user data\n"); + const result = runTool(PACKAGER, ["--source", src, "--out", out, "--skip-verify", "--no-tar"]); + const preserved = existsSync(join(out, "precious.txt")); + expectFail("A2 refuse nonempty output (content preserved)", { ...result, status: preserved ? result.status : 99 }, "not empty", sandbox); + } + + // A3: refuse output inside the source tree. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-a3-")); + const src = makeSyntheticSource(sandbox); + const result = runTool(PACKAGER, ["--source", src, "--out", join(src, "inside"), "--skip-verify", "--no-tar"]); + expectFail("A3 refuse output inside source", result, "lies inside the source tree", sandbox); + } + + // A4: output nested under a COMMON ancestor of the source is allowed (guard + // against an over-broad disjointness rule); output nested under the source + // itself through not-yet-existing directories is refused. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-a4-")); + try { + const base = join(sandbox, "x"); + const src = makeSyntheticSourceIn(sandbox, base); // x/src + const okOut = join(base, "out"); // sibling of src under x — allowed + const ok = runTool(PACKAGER, ["--source", src, "--out", okOut, "--skip-verify", "--no-tar"]); + const nested = runTool(PACKAGER, ["--source", src, "--out", join(src, "sub", "deep", "out"), "--skip-verify", "--no-tar"]); + const passed = ok.status === 0 && nested.status !== 0 && `${nested.stderr}`.includes("lies inside the source tree"); + record("A4 common-ancestor output allowed; nested-in-source refused", passed, passed ? "" : `ok=${ok.status} nested=${nested.status}: ${nested.stderr.slice(0, 160)}`); + } finally { + rmSync(sandbox, { recursive: true, force: true }); + } + } + + // A5: refuse symlinked output directory. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-a5-")); + const src = makeSyntheticSource(sandbox); + const realOut = join(sandbox, "real-out"); + mkdirSync(realOut, { recursive: true }); + symlinkSync(realOut, join(sandbox, "link-out")); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "link-out"), "--skip-verify", "--no-tar"]); + expectFail("A5 refuse symlinked output", result, "symlink", sandbox); + } + + // A6: refuse symlinked ancestor (path redirection). + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-a6-")); + const src = makeSyntheticSource(sandbox); + const realDir = join(sandbox, "real-parent"); + mkdirSync(realDir, { recursive: true }); + symlinkSync(realDir, join(sandbox, "link-parent")); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "link-parent", "out"), "--skip-verify", "--no-tar"]); + expectFail("A6 refuse symlinked ancestor", result, "symlink", sandbox); + } + + // A7: refuse preexisting archive/checksum/summary targets. + for (const suffix of [".tar.gz", ".tar.gz.sha256", ".build-summary.json"]) { + const sandbox = mkdtempSync(join(tmpdir(), `pi-pkg-a7-`)); + const src = makeSyntheticSource(sandbox); + writeFileSync(join(sandbox, `out${suffix}`), "stale target\n"); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "out"), "--skip-verify"]); + expectFail(`A7 refuse preexisting ${suffix} target`, result, "preexisting output target", sandbox); + } +} + +function makeSyntheticSourceIn(sandbox, base) { + // Build a synthetic source at base/src (helper for the ancestor test). + const src = join(base, "src"); + mkdirSync(join(src, "packages", "coding-agent", "dist"), { recursive: true }); + writeJson(join(src, "package.json"), { name: "pi-monorepo", version: "0.0.0" }); + writeJson(join(src, "packages", "coding-agent", "package.json"), { name: "@earendil-works/pi-coding-agent", version: "1.1.0", license: "MIT" }); + writeFileSync(join(src, "packages", "coding-agent", "dist", "index.js"), 'export const VERSION = "1.1.0-test";\n'); + writeFileSync(join(src, "packages", "coding-agent", "dist", "cli.js"), "export {};\n"); + writeFileSync(join(src, "packages", "coding-agent", "dist", "rpc-entry.js"), "export {};\n"); + writeFileSync(join(src, "LICENSE"), "MIT test license\n"); + return src; +} + +function testClosurePolicy() { + for (const prefix of ['"./"', "`./`"]) { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-prefix-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "computed-prefix";\n' }); + addThirdParty(src, "computed-prefix", `export function load(name) { return import(${prefix} + name); }\n`); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "out"), "--skip-verify", "--no-tar"]); + expectFail(`${prefix} computed import requires dynamic audit`, result, "unaudited nonliteral dynamic", sandbox); + } + // B1: missing relative import (first-party) is a hard failure. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-b1-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "./missing.js";\nexport const VERSION = "1.1.0-test";\n' }); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "out"), "--skip-verify", "--no-tar"]); + expectFail("B1 missing relative runtime edge refused", result, "does not resolve to a file", sandbox); + } + + // B2: nonliteral dynamic import in an AUDITED package passes and is recorded. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-b2-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "jiti";\nexport const VERSION = "1.1.0-test";\n' }); + addThirdParty(src, "jiti", "export function jit(id) { return require(id); }\n"); + const out = join(sandbox, "out"); + const result = runTool(PACKAGER, ["--source", src, "--out", out, "--skip-verify", "--no-tar"]); + let audited = false; + if (result.status === 0) { + const manifest = JSON.parse(readFileSync(join(out, "manifest.json"), "utf8")); + audited = (manifest.auditedDynamicImports ?? []).some((recorded) => recorded.package === "jiti"); + } + record("B2 audited nonliteral dependency passes + recorded", result.status === 0 && audited, result.status === 0 && audited ? "" : `status=${result.status} audited=${audited}: ${result.stderr.slice(0, 200)}`); + rmSync(sandbox, { recursive: true, force: true }); + } + + // B3: nonliteral dynamic import in an UNAUDITED package fails. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-b3-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "mystery-pkg";\nexport const VERSION = "1.1.0-test";\n' }); + addThirdParty(src, "mystery-pkg", "export function load(id) { return require(id); }\n"); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "out"), "--skip-verify", "--no-tar"]); + expectFail("B3 unaudited nonliteral dependency refused", result, "unaudited nonliteral dynamic import site", sandbox); + } + + // B4: classified optional import (third-party) passes with a recorded classification. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-b4-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "debug";\nexport const VERSION = "1.1.0-test";\n' }); + addThirdParty(src, "debug", 'import "supports-color";\nexport default {};\n'); + const out = join(sandbox, "out"); + const result = runTool(PACKAGER, ["--source", src, "--out", out, "--skip-verify", "--no-tar"]); + let classified = false; + if (result.status === 0) { + const manifest = JSON.parse(readFileSync(join(out, "manifest.json"), "utf8")); + classified = (manifest.warnings ?? []).some((warning) => warning.spec === "supports-color" && warning.classification === "optional-dependency"); + } + record("B4 classified optional import passes + recorded", result.status === 0 && classified, result.status === 0 && classified ? "" : `status=${result.status} classified=${classified}: ${result.stderr.slice(0, 200)}`); + rmSync(sandbox, { recursive: true, force: true }); + } + + // B5: unclassified unresolved third-party import fails. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-b5-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "weird-pkg";\nexport const VERSION = "1.1.0-test";\n' }); + addThirdParty(src, "weird-pkg", 'import "not-installed-anywhere";\nexport default {};\n'); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "out"), "--skip-verify", "--no-tar"]); + expectFail("B5 unclassified unresolved import refused", result, "unclassified", sandbox); + } + + // B6: wrong-arch-only native dependency is refused (x64-only when targeting arm64). + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-b6-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "native-only-x64";\nexport const VERSION = "1.1.0-test";\n' }); + addThirdParty(src, "native-only-x64", "export const native = true;\n", { "prebuilds/x64.node": { buffer: elfHeader(EM_X64) } }); + const result = runTool(PACKAGER, ["--source", src, "--out", join(sandbox, "out"), "--skip-verify", "--no-tar"]); + expectFail("B6 wrong-arch-only native refused (x64-only, target arm64)", result, "ships no arm64", sandbox); + } + + // B7: target-arch native with host sibling passes and is inventoried. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-pkg-b7-")); + const src = makeSyntheticSource(sandbox, { distIndex: 'import "native-dual";\nexport const VERSION = "1.1.0-test";\n' }); + addThirdParty(src, "native-dual", "export const native = true;\n", { + "prebuilds/arm64.node": { buffer: elfHeader(EM_ARM64) }, + "prebuilds/x64.node": { buffer: elfHeader(EM_X64) }, + }); + const out = join(sandbox, "out"); + const result = runTool(PACKAGER, ["--source", src, "--out", out, "--skip-verify", "--no-tar"]); + let inventoried = false; + if (result.status === 0) { + const manifest = JSON.parse(readFileSync(join(out, "manifest.json"), "utf8")); + inventoried = + manifest.nativeArtifacts.some((artifact) => artifact.path.endsWith("arm64.node") && artifact.coverage === "target") && + manifest.nativeArtifacts.some((artifact) => artifact.path.endsWith("x64.node") && String(artifact.coverage).includes("sibling")); + } + record("B7 target+host native inventory passes with coverage labels", result.status === 0 && inventoried, result.status === 0 && inventoried ? "" : `status=${result.status} inventoried=${inventoried}: ${result.stderr.slice(0, 200)}`); + rmSync(sandbox, { recursive: true, force: true }); + } +} + +// --------------------------------------------------------------------------- +// Group C: verify-bundle manifest rejection BEFORE side-effect import +// --------------------------------------------------------------------------- + +function makeFakeBundle(root, { tamper } = {}) { + // Minimal bundle whose SDK entry has an import side effect (marker file). + const bundle = join(root, "bundle"); + const sdkDir = join(bundle, "node_modules", "@earendil-works", "pi-coding-agent", "dist"); + mkdirSync(sdkDir, { recursive: true }); + writeFileSync( + join(sdkDir, "index.js"), + [ + 'import { writeFileSync, mkdirSync } from "node:fs";', + 'import { dirname, join } from "node:path";', + 'import { fileURLToPath } from "node:url";', + '// Side effect proving this module was imported.', + 'const bundleRoot = join(dirname(fileURLToPath(import.meta.url)), "..", "..", "..", "..");', + 'writeFileSync(join(bundleRoot, "SDK-IMPORT-EXECUTED"), "yes");', + 'export const VERSION = "1.1.0-test";', + "", + ].join("\n"), + ); + writeJson(join(bundle, "node_modules", "@earendil-works", "pi-coding-agent", "package.json"), { + name: "@earendil-works/pi-coding-agent", + version: "1.1.0", + }); + writeJson(join(bundle, "manifest.json"), { generator: "test", target: { arch: "arm64" } }); + + const files = []; + const walk = (dir) => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const path = join(dir, entry.name); + if (entry.isDirectory()) walk(path); + else files.push(relative(bundle, path)); + } + }; + walk(bundle); + const sha = (file) => createHash("sha256").update(readFileSync(file)).digest("hex"); + const baseLines = ["# test manifest", ...files.map((f) => `${sha(join(bundle, f))} ${f}`)]; + const extraLines = tamper ? tamper(bundle, sha, baseLines) || [] : []; + writeFileSync(join(bundle, "manifest.txt"), [...baseLines, ...extraLines, ""].join("\n")); + return bundle; +} + +function testVerifierRejection() { + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-verify-root-fifo-")); + try { + const bundle = makeFakeBundle(sandbox, {}); + const manifest = join(bundle, "manifest.txt"); + rmSync(manifest); + const mk = spawnSync("mkfifo", [manifest]); + if (mk.status !== 0) throw new Error("mkfifo required for manifest root regression"); + const result = spawnSync(process.execPath, [VERIFIER, "--bundle", bundle], { encoding: "utf8", timeout: 3000 }); + record("manifest root FIFO rejects promptly before SDK import", result.status === 1 && !result.error + && result.stderr.includes("bounded regular file") && !existsSync(join(bundle, "SDK-IMPORT-EXECUTED")), result.stderr || String(result.error)); + } finally { rmSync(sandbox, { recursive: true, force: true }); } + } + const cases = [ + { + name: "C1 traversal entry refused before import", + tamper: (bundle, sha) => { + writeFileSync(join(bundle, "..", "outside.txt"), "outside\n"); + return [`${sha(join(bundle, "..", "outside.txt"))} ../outside.txt`]; + }, + }, + { + name: "C2 absolute entry refused before import", + tamper: (bundle, sha) => [`${sha("/etc/passwd")} /etc/passwd`], + }, + { + name: "C3 duplicate entry refused before import", + tamper: (bundle, sha) => [`${sha(join(bundle, "manifest.json"))} manifest.json`], + }, + { + name: "C4 symlink in bundle refused before import", + tamper: (bundle) => symlinkSync(join(bundle, "manifest.json"), join(bundle, "node_modules", "alias.json")), + }, + { + name: "C5 unlisted file refused before import", + tamper: () => {}, + afterManifest: (bundle) => writeFileSync(join(bundle, "node_modules", "sneaky.js"), "export {};\n"), + }, + { + name: "C6 hash mismatch refused before import", + tamper: () => {}, + afterManifest: (bundle) => { + // Corrupt a file AFTER the manifest was written. + const path = join(bundle, "node_modules", "@earendil-works", "pi-coding-agent", "package.json"); + writeFileSync(path, `${readFileSync(path, "utf8")}\n// tampered\n`); + }, + }, + { + name: "C7 missing listed file refused before import", + tamper: () => {}, + afterManifest: (bundle) => rmSync(join(bundle, "manifest.json")), + }, + ]; + for (const testCase of cases) { + const sandbox = mkdtempSync(join(tmpdir(), "pi-verify-c-")); + try { + const bundle = makeFakeBundle(sandbox, { tamper: testCase.tamper }); + if (testCase.afterManifest) testCase.afterManifest(bundle); + const marker = join(bundle, "SDK-IMPORT-EXECUTED"); + const result = runTool(VERIFIER, ["--bundle", bundle]); + const rejected = result.status !== 0; + const noImport = !existsSync(marker); + record(testCase.name, rejected && noImport, rejected && noImport ? `${(result.stderr || result.stdout).split("\n")[0].slice(0, 90)}` : `status=${result.status} marker=${existsSync(marker)}`); + } finally { + rmSync(sandbox, { recursive: true, force: true }); + } + } + + // C8: nonregular file (FIFO) refused before import. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-verify-c8-")); + try { + const bundle = makeFakeBundle(sandbox, {}); + const fifo = join(bundle, "node_modules", "pipe"); + const mk = spawnSync("mkfifo", [fifo]); + let passed; + let detail = ""; + if (mk.status !== 0) { + passed = true; + detail = "mkfifo unavailable on this host; skipped (symlink case covers non-regular rejection)"; + } else { + const marker = join(bundle, "SDK-IMPORT-EXECUTED"); + const result = runTool(VERIFIER, ["--bundle", bundle]); + passed = result.status !== 0 && !existsSync(marker); + detail = passed ? "" : `status=${result.status} marker=${existsSync(marker)}`; + } + record("C8 nonregular file (fifo) refused before import", passed, detail); + } finally { + rmSync(sandbox, { recursive: true, force: true }); + } + } + + // C9: aliased (symlinked) bundle path refused. + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-verify-c9-")); + try { + const bundle = makeFakeBundle(sandbox, {}); + symlinkSync(bundle, join(sandbox, "alias-bundle")); + const result = runTool(VERIFIER, ["--bundle", join(sandbox, "alias-bundle")]); + const marker = join(bundle, "SDK-IMPORT-EXECUTED"); + record("C9 aliased bundle path refused before import", result.status !== 0 && !existsSync(marker), (result.stderr || "").split("\n")[0].slice(0, 90)); + } finally { + rmSync(sandbox, { recursive: true, force: true }); + } + } + + // C10: clean manifest → the SDK IS imported (order proven: manifest first, import second). + { + const sandbox = mkdtempSync(join(tmpdir(), "pi-verify-c10-")); + try { + const bundle = makeFakeBundle(sandbox, {}); + const marker = join(bundle, "SDK-IMPORT-EXECUTED"); + const result = runTool(VERIFIER, ["--bundle", bundle]); + const imported = existsSync(marker); + const manifestFirst = (result.stdout + result.stderr).includes("manifest verified FIRST"); + record("C10 clean manifest reaches SDK import (marker written after manifest ok)", imported && manifestFirst, `imported=${imported} manifestFirst=${manifestFirst} status=${result.status}`); + } finally { + rmSync(sandbox, { recursive: true, force: true }); + } + } +} + +// --------------------------------------------------------------------------- + +function main() { + console.log(`sdk-package-tests: node ${process.version}, host ${process.platform}/${process.arch}`); + testPackagerSafety(); + testClosurePolicy(); + testVerifierRejection(); + const failed = results.filter((entry) => !entry.passed); + console.log(`\n${results.length - failed.length}/${results.length} tests passed`); + if (failed.length > 0) { + for (const entry of failed) console.error(`FAILED: ${entry.name}${entry.detail ? ` — ${entry.detail}` : ""}`); + process.exit(1); + } +} + +main(); diff --git a/portable/raspberry-pi/sdk-runtime.md b/portable/raspberry-pi/sdk-runtime.md new file mode 100644 index 0000000..1d10e58 --- /dev/null +++ b/portable/raspberry-pi/sdk-runtime.md @@ -0,0 +1,141 @@ +# Raspberry Pi (ARM64) SDK runtime bundle + +Tools to build and verify a small, transferable runtime bundle of the vendored +Pi SDK (`@earendil-works/pi-coding-agent` 1.1.x) for deployment on an ARM64 +(aarch64) Linux guest. + +- `package-sdk.mjs` — build the bundle from the built vendored source tree. +- `verify-bundle.mjs` — verify a bundle (ships inside it as + `verify/verify-bundle.mjs`); the receiving machine reruns the exact checks. +- `sdk-package-tests.mjs` — synthetic-tree tests (no user data): output + safety, closure policy, and manifest-rejection-before-import ordering. + +## Build + +```sh +node package-sdk.mjs --source /vendor/pi --out --target-arch arm64 +``` + +No network access, no installs, no guest interaction, and **nothing is ever +deleted**: the packager refuses a nonempty output, symlinked outputs or +ancestors, output/source overlap (either direction), and preexisting +`.tar.gz` / `.sha256` / `.build-summary.json` targets. There is no +pruning of older bundles — retire outputs by hand. Failed runs leave partial +output in place; reruns refuse it (choose a fresh directory). + +Outputs (all next to each other): + +- `/` — the bundle directory (regular files and directories only). +- `.tar.gz` + `.tar.gz.sha256` — the transfer archive and its + **transfer digest**. +- `.build-summary.json` — external build record (verification result, + archive digest, open gates). Never inside the bundle, so the archive + contains exactly the hashed content. + +## Bundle format + +- **No symlinks at all** (simplest portable format). Any symlink found while + copying or verifying is a hard error. +- `manifest.txt` is the integrity root: the sha256 of **every** file in the + bundle except itself (including `manifest.json` and + `THIRD_PARTY_NOTICES.md`). Files must be relative POSIX paths without + absolute/traversal/duplicate/alias components; the bundle must contain + nothing unlisted and nothing missing. +- A manifest checksum is **integrity relative to the separately trusted + transfer digest** (the `.tar.gz.sha256` verified over a trusted channel); + it is not authenticity of the source tree. +- `manifest.json` carries metadata only: target, provenance (source + + lockfile digests, exact package versions, generator digests — the checkout + may not be a git repo), package inventory with declared licenses (metadata, + not notice provenance; shipped license files are preserved in place), + classified closure warnings, audited dynamic-import sites, and the native + artifact inventory. + +## Closure policy + +The static scan traces literal `import`/`require`/`require.resolve`/`import()` +specifiers (comment stripping is regex-literal aware) and: + +- fails on **missing relative imports** (first-party) — never silently ignored; +- fails on unresolved third-party imports **unless classified** in + `OPTIONAL_IMPORTS` with a concrete audited reason (optional native deps such + as `kerberos`, `bufferutil`, `utf-8-validate`, AWS CRT signer deps; + generator-template false positives inside `jiti`); +- fails on **nonliteral (computed) `require()`/`import()` sites** unless the + owning package is audited in `DYNAMIC_IMPORT_AUDIT`; +- respects package `exports` maps (a subpath not exposed by `exports` is an + error, matching Node runtime behavior). + +This is a heuristic static closure, not complete static analysis. Runtime +verification covers only the exercised paths (SDK import, quickjs +instantiation, the write tool). **No universal SDK completeness is claimed.** + +## Native policy + +- Every `.node` ELF must be target-arch, or host-arch **with a target-arch + sibling in the same package**. Wrong-arch-only dependencies are hard + failures with a concrete blocker message. +- WASM payloads are architecture independent (photon, quickjs and extensions). +- Host verification actually `dlopen`s native bindings whose architecture + matches the host. Target-arch bindings (e.g. the arm64 `pi-tui` prebuild on + an x64 host) are an **explicit open gate** and are never claimed from the + host. The `pi-tui` binding is a clipboard helper (X11); if a target lacks + the display stack, report that gate separately. + +## Verification + +```sh +node verify/verify-bundle.mjs # from the bundle root +``` + +Order (fail closed): + +1. **Environment isolation** before any bundle code import: the process env is + reduced to an allowlist (`PATH`, `LANG`, `LC_*`, `TZ`, `TERM`, plus + disposable `HOME`/`TMPDIR` and `PI_OFFLINE=1`); `NODE_OPTIONS`, `PI_*` + and everything else are dropped. Discovery canaries (extensions, skills, + prompts, themes, context files, `auth.json`, `models.json`) are planted in + the disposable HOME and asserted **not executed**. +2. **Complete manifest check before any runtime import** (see format above). +3. SDK import probe, native header + host-arch load checks, quickjs WASM + instantiation, and a scripted agent session that executes the real write + tool under the isolated agent-worker recipe + (`SettingsManager.inMemory` with retry/compaction/cacheWarming off, + `DefaultResourceLoader` with `noExtensions/noSkills/noPromptTemplates/ + noThemes/noContextFiles`, explicit system prompt, in-memory + auth/models/session, `refreshOnCreate: false`, `allowModelNetwork: false`). + +## ARM handoff (parent-coordinated; these tools never touch the guest) + +1. Verify the trusted transfer digest on the sending side: + `sha256sum -c sdk-reviewed-bundle.tar.gz.sha256`. +2. Inspect archive member paths and link types (regular files only): + `tar -tvzf sdk-reviewed-bundle.tar.gz`. +3. Transfer only the reviewed archive to a **new guest temp directory**; never + overwrite an existing SDK/deployment. +4. On the guest (expect `uname -m` = `aarch64`, `process.arch` = `arm64`, + Node 24 ARM runtime): extract, then run + `node verify/verify-bundle.mjs` from the bundle root. Record + `uname -m`, `process.platform/arch/version`, archive/source hashes, and + the verifier output. The arm64 `pi-tui` binding load is the open gate this + closes. +5. Limitations to record: host checks are x86 evidence only; the scripted + write-tool session proves that path, not universal SDK completeness; no + live inference, wallet-credential, or Tor qualification follows from these + checks; a vendored-relative deployment path (e.g. the workspace + `agent-worker`) is not automatically integrated by standalone bundle + proof. + +## Tests + +```sh +node sdk-package-tests.mjs +``` + +26 synthetic-tree tests: sentinel/sibling survival, nonempty/inside-source/ +symlink/preexisting-target refusal, missing runtime edges, audited vs +unaudited nonliteral dependencies, classified vs unclassified optional +imports, wrong-arch-only native refusal, and manifest rejection **before** +side-effect import (traversal/absolute/duplicate/symlink/unlisted/hash +mismatch/missing/nonregular/alias), plus order proof that a clean manifest +reaches the SDK import. diff --git a/portable/raspberry-pi/verify-bundle.mjs b/portable/raspberry-pi/verify-bundle.mjs new file mode 100644 index 0000000..c8f1e4b --- /dev/null +++ b/portable/raspberry-pi/verify-bundle.mjs @@ -0,0 +1,585 @@ +#!/usr/bin/env node +/** + * Verify a Pi SDK runtime bundle produced by portable/raspberry-pi/package-sdk.mjs. + * + * The same script ships inside the bundle (verify/verify-bundle.mjs) so the + * receiving machine can repeat the exact checks after transfer: + * + * node verify/verify-bundle.mjs (from the bundle root) + * + * ORDER OF CHECKS (fail closed, manifest first): + * 1. Environment isolation: the process env is reduced to an allowlist and + * HOME/TMPDIR are replaced with disposable directories BEFORE any bundle + * code is imported. Discovery canaries are planted in the disposable HOME. + * 2. Complete manifest check BEFORE any runtime import of bundle code: + * every entry is a safe relative path (no absolute/traversal/duplicate/ + * alias), the bundle contains only regular files and directories (no + * symlinks, no special files), nothing is missing, nothing unlisted, and + * every hash matches. manifest.txt is the integrity root; it is integrity + * relative to the separately trusted transfer digest (.tar.gz.sha256 + * verified over a trusted channel), NOT authenticity. + * 3. Only then: SDK import probe, native artifact checks (headers for every + * artifact, actual dlopen of host-arch bindings; target-arch bindings are + * an explicit open gate when the host differs), quickjs WASM + * instantiation, and a scripted agent session driving the real write tool + * under the isolated agent-worker recipe with canary assertions. + * + * Scope: these checks prove manifest integrity, that the imported SDK entry + * points load, that quickjs instantiates, and that the write tool executes. + * They do NOT prove universal SDK completeness, and a successful host run is + * never proof for the target architecture. + */ +import { createHash } from "node:crypto"; +import { openSync, closeSync, fstatSync, constants, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync, lstatSync, realpathSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, relative, resolve, sep } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url)); + +const ENV_ALLOWLIST = ["PATH", "LANG", "LC_ALL", "LC_CTYPE", "TZ", "TERM"]; +const SYSTEM_PROMPT = "You are a bundle verification assistant. Follow the operator's instruction exactly."; + +// --------------------------------------------------------------------------- +// Failure handling with cleanup of disposable directories +// --------------------------------------------------------------------------- + +const cleanupFns = []; + +function cleanup() { + while (cleanupFns.length > 0) { + const fn = cleanupFns.pop(); + try { + fn(); + } catch { + // best effort + } + } +} + +function fail(message) { + cleanup(); + console.error(`verify-bundle: FAIL: ${message}`); + process.exit(1); +} + +function ok(message) { + console.log(`verify-bundle: ok: ${message}`); +} + +function parseArgs(argv) { + const options = { bundle: "" }; + for (let i = 0; i < argv.length; i++) { + if (argv[i] === "--bundle") options.bundle = resolve(argv[++i]); + else if (argv[i] === "--check-manifest") { + // Kept for compatibility: the manifest is ALWAYS checked first. + } else fail(`unknown argument ${argv[i]}`); + } + if (!options.bundle) options.bundle = resolve(SCRIPT_DIR, ".."); + return options; +} + +// --------------------------------------------------------------------------- +// 0. Environment isolation (before any bundle code is imported) +// --------------------------------------------------------------------------- + +function isolateEnvironment() { + const isolated = mkdtempSync(join(tmpdir(), "pi-verify-env-")); + const home = join(isolated, "home"); + const tmp = join(isolated, "tmp"); + const markerDir = join(isolated, "markers"); + mkdirSync(home, { recursive: true }); + mkdirSync(tmp, { recursive: true }); + mkdirSync(markerDir, { recursive: true }); + cleanupFns.push(() => rmSync(isolated, { recursive: true, force: true })); + + // Plant discovery canaries in the disposable HOME. If any discovery path + // executed despite the isolation flags, markers would appear and the + // canary assertions below would fail. + const agentDir = join(home, ".pi", "agent"); + for (const dir of ["extensions", "skills/canary-skill", "prompts", "themes"]) { + mkdirSync(join(agentDir, dir), { recursive: true }); + } + writeFileSync(join(agentDir, "auth.json"), `${JSON.stringify({ "canary-provider": { type: "api_key", key: "canary-key" } }, null, 2)}\n`); + writeFileSync( + join(agentDir, "models.json"), + `${JSON.stringify( + { providers: { "canary-provider": { baseUrl: "http://127.0.0.1:9", api: "anthropic-messages", models: [{ id: "canary-model", name: "Canary Model" }] } } }, + null, + 2, + )}\n`, + ); + writeFileSync(join(agentDir, "SYSTEM.md"), "CANARY-GLOBAL-SYSTEM-PROMPT must never appear in the session system prompt.\n"); + writeFileSync( + join(agentDir, "extensions", "canary-ext.mjs"), + [ + 'import { writeFileSync } from "node:fs";', + '// If discovery imports this file, the marker proves it.', + 'writeFileSync(process.env.PI_VERIFY_CANARY_DIR + "/extension-executed", "yes");', + "export default {};", + "", + ].join("\n"), + ); + writeFileSync(join(agentDir, "skills", "canary-skill", "SKILL.md"), "---\nname: canary-skill\ndescription: canary\n---\nCANARY SKILL\n"); + writeFileSync(join(agentDir, "prompts", "canary-prompt.md"), "CANARY PROMPT TEMPLATE\n"); + writeFileSync(join(agentDir, "themes", "canary-theme.json"), '{"name":"canary-theme","CANARY":true}\n'); + + const clean = {}; + for (const key of ENV_ALLOWLIST) { + if (process.env[key] !== undefined) clean[key] = process.env[key]; + } + clean.HOME = home; + clean.TMPDIR = tmp; + clean.PI_OFFLINE = "1"; // model network disabled at the runtime level too + clean.PI_VERIFY_CANARY_DIR = markerDir; + for (const key of Object.keys(process.env)) delete process.env[key]; + Object.assign(process.env, clean); + + return { home, tmp, markerDir }; +} + +// --------------------------------------------------------------------------- +// 1. Manifest check (complete, strict, BEFORE any runtime import) +// --------------------------------------------------------------------------- + +const ELF_MACHINE_TO_ARCH = { 3: "ia32", 40: "arm", 62: "x64", 183: "arm64", 243: "riscv64" }; + +function inspectBinary(file) { + const data = readFileSync(file); + if (data.length >= 20 && data[0] === 0x7f && data[1] === 0x45 && data[2] === 0x4c && data[3] === 0x46) { + return { format: "elf", bits: data[4] === 2 ? 64 : 32, arch: ELF_MACHINE_TO_ARCH[data.readUInt16LE(18)] || "unknown" }; + } + if (data.length >= 4 && data[0] === 0x00 && data[1] === 0x61 && data[2] === 0x73 && data[3] === 0x6d) { + return { format: "wasm", arch: "wasm32 (portable)" }; + } + return { format: "unknown", arch: "?" }; +} + +function checkManifest(bundleDir) { + // The bundle root itself must not be a symlink/path alias. + const realBundle = realpathSync(bundleDir); + if (realBundle !== resolve(bundleDir)) { + fail(`bundle path is an alias (symlink?): ${bundleDir} -> ${realBundle}; pass the real path`); + } + const manifestPath = join(bundleDir, "manifest.txt"); + if (!existsSync(manifestPath)) fail("manifest.txt missing"); + // Open without following links or blocking on FIFOs, then validate the + // descriptor before reading. The manifest root needs the same type gate. + const manifestFd = openSync(manifestPath, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + let manifestText; + try { + const info = fstatSync(manifestFd); + if (!info.isFile() || info.size > 16 * 1024 * 1024) fail("manifest.txt must be a bounded regular file"); + manifestText = readFileSync(manifestFd, "utf8"); + } finally { closeSync(manifestFd); } + const entries = new Map(); + for (const line of manifestText.split("\n")) { + if (line.startsWith("#") || line.trim() === "") continue; + const match = /^([0-9a-f]{64}) (.+)$/.exec(line); + if (!match) fail(`malformed manifest line: ${JSON.stringify(line)}`); + const [, sha, rawPath] = match; + if (rawPath.startsWith("/") || /^[A-Za-z]:[\\/]/.test(rawPath)) fail(`manifest entry is absolute: ${rawPath}`); + if (rawPath.includes("\\")) fail(`manifest entry contains a backslash (non-portable): ${rawPath}`); + const segments = rawPath.split("/"); + if (segments.some((segment) => segment === "" || segment === "." || segment === "..")) { + fail(`manifest entry is empty/traversal: ${rawPath}`); + } + if (rawPath === "manifest.txt") fail("manifest.txt must not list itself (it is the integrity root)"); + if (entries.has(rawPath)) fail(`duplicate manifest entry: ${rawPath}`); + const file = resolve(bundleDir, rawPath); + if (!(file === bundleDir || file.startsWith(bundleDir + sep))) fail(`manifest entry escapes the bundle: ${rawPath}`); + entries.set(rawPath, sha); + } + if (entries.size === 0) fail("manifest.txt lists no files"); + + // Walk the bundle: regular files and directories only. + const diskFiles = new Set(["manifest.txt"]); + const walk = (dir) => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const path = join(dir, entry.name); + const rel = relative(bundleDir, path); + if (entry.isSymbolicLink()) fail(`bundle contains a symlink (format forbids symlinks): ${rel}`); + if (entry.isDirectory()) { + walk(path); + continue; + } + if (!entry.isFile()) fail(`bundle contains a non-regular file: ${rel}`); + diskFiles.add(rel); + } + }; + walk(bundleDir); + + for (const [rel, expected] of entries) { + const file = join(bundleDir, rel); + if (!existsSync(file)) fail(`manifest lists missing file: ${rel}`); + let stat; + try { + stat = lstatSync(file); + } catch { + fail(`cannot stat manifest entry: ${rel}`); + } + if (!stat.isFile()) fail(`manifest entry is not a regular file: ${rel}`); + if (realpathSync(file) !== file) fail(`manifest entry is a path alias (realpath differs): ${rel}`); + const sha256 = createHash("sha256").update(readFileSync(file)).digest("hex"); + if (sha256 !== expected) fail(`checksum mismatch: ${rel}`); + } + for (const rel of diskFiles) { + if (rel === "manifest.txt") continue; // the integrity root cannot hash itself + if (!entries.has(rel)) fail(`bundle contains a file not listed in manifest.txt: ${rel}`); + } + + // manifest.json must exist and parse (it is hashed content, checked above). + const manifestJsonPath = join(bundleDir, "manifest.json"); + if (!entries.has("manifest.json")) fail("manifest.txt does not cover manifest.json"); + try { + const meta = JSON.parse(readFileSync(manifestJsonPath, "utf8")); + if (typeof meta.generator !== "string" || !meta.target?.arch) { + fail("manifest.json is missing generator/target metadata"); + } + } catch (error) { + fail(`manifest.json is unreadable: ${error.message}`); + } + ok(`manifest verified FIRST: ${entries.size} files hashed and matched, no symlinks, no missing/unlisted/absolute/traversal/duplicate entries`); +} + +// --------------------------------------------------------------------------- +// 2. SDK probe (only after the manifest check passed) +// --------------------------------------------------------------------------- + +async function importFromBundle(bundleDir, relativePath) { + const file = join(bundleDir, "node_modules", "@earendil-works", relativePath); + if (!existsSync(file)) fail(`bundle is missing ${relative(bundleDir, file)}`); + return import(pathToFileURL(file).href); +} + +async function probeSdk(bundleDir) { + const sdk = await importFromBundle(bundleDir, "pi-coding-agent/dist/index.js"); + for (const name of ["VERSION", "AgentSession", "SessionManager", "SettingsManager", "DefaultResourceLoader", "createCodingTools", "createWriteTool", "ModelRuntime", "runPrintMode"]) { + if (!(name in sdk)) fail(`SDK entry does not export ${name}`); + } + ok(`SDK loads from bundle: @earendil-works/pi-coding-agent@${sdk.VERSION}`); + + const ai = await importFromBundle(bundleDir, "pi-ai/dist/index.js"); + if (typeof ai.createAssistantMessageEventStream !== "function") fail("pi-ai does not export createAssistantMessageEventStream"); + const agentCore = await importFromBundle(bundleDir, "pi-agent-core/dist/index.js"); + if (typeof agentCore.Agent !== "function") fail("pi-agent-core does not export Agent"); + ok("pi-ai and pi-agent-core load from bundle"); + return { sdk, ai, agentCore }; +} + +// --------------------------------------------------------------------------- +// 3. Native and WASM artifacts +// --------------------------------------------------------------------------- + +/** Package root (bundle-relative) of a file inside node_modules. */ +function packageRootOfRelative(relPath) { + const segments = relPath.split("/"); + const idx = segments.lastIndexOf("node_modules"); + if (idx === -1 || idx === segments.length - 1) return relPath; + const rest = segments.slice(idx + 1); + const root = rest[0].startsWith("@") && rest.length > 1 ? [rest[0], rest[1]] : [rest[0]]; + return [...segments.slice(0, idx + 1), ...root].join("/"); +} + +async function checkArtifacts(bundleDir) { + const manifest = JSON.parse(readFileSync(join(bundleDir, "manifest.json"), "utf8")); + const targetArch = manifest.target?.arch ?? fail("manifest.json has no target.arch"); + const hostArch = process.arch; + const artifacts = manifest.nativeArtifacts ?? []; + const elfArchesByPackage = new Map(); + for (const artifact of artifacts) { + const info = inspectBinary(join(bundleDir, artifact.path)); + if (info.format !== artifact.format || info.arch !== artifact.arch) { + fail(`native artifact ${artifact.path} does not match the manifest claim (${JSON.stringify(info)} vs ${JSON.stringify(artifact)})`); + } + if (info.format === "elf") { + const root = packageRootOfRelative(artifact.path); + if (!elfArchesByPackage.has(root)) elfArchesByPackage.set(root, new Set()); + elfArchesByPackage.get(root).add(info.arch); + } + } + // Target-coverage rule: every non-target ELF package must also ship a + // target-arch ELF (this revalidates the packager's rule on the target). + for (const artifact of artifacts) { + if (artifact.format !== "elf") continue; + if (artifact.arch === targetArch) continue; + const root = packageRootOfRelative(artifact.path); + if (!elfArchesByPackage.get(root)?.has(targetArch)) { + fail(`native artifact ${artifact.path} is ${artifact.arch} and package ${root} ships no ${targetArch} variant (wrong-arch-only dependency)`); + } + } + // pi-tui prebuilds: header check for each bundled arch, plus an ACTUAL + // binding load for prebuilds matching this host. Non-host prebuilds are an + // explicit open gate, never claimed from here. + const prebuildBase = join("node_modules", "@earendil-works", "pi-tui", "native", "linux", "prebuilds"); + for (const artifact of artifacts) { + if (!artifact.path.startsWith(prebuildBase) || !artifact.path.endsWith(".node")) continue; + const arch = artifact.arch; + ok(`pi-tui prebuild ${relative(prebuildBase, dirname(artifact.path))}: ELF header validated (${arch})`); + if (arch === hostArch) { + try { + const { createRequire } = await import("node:module"); + const req = createRequire(import.meta.url); + const binding = req(join(bundleDir, artifact.path)); + ok(`pi-tui ${arch} native binding ACTUALLY LOADED on this host (exports: ${Object.keys(binding).join(", ") || "none"})`); + } catch (error) { + fail(`pi-tui ${arch} native binding failed to load on this ${hostArch} host: ${error.message}`); + } + } else { + console.log(`verify-bundle: gate: pi-tui ${arch} binding NOT loadable on host ${hostArch}; open gate for on-target verification`); + } + } + // WASM payloads are validated by header here and instantiated below. + const wasmTargets = [ + ["@silvia-odwyer/photon-node/photon_rs_bg.wasm", "photon image conversion"], + ["quickjs-wasi/quickjs.wasm", "codemode quickjs runtime"], + ]; + for (const [path, label] of wasmTargets) { + const file = join(bundleDir, "node_modules", path); + if (!existsSync(file)) fail(`missing ${label} payload: ${path}`); + const info = inspectBinary(file); + if (info.format !== "wasm") fail(`${label} payload is not WebAssembly: ${path}`); + ok(`${label}: WebAssembly payload present (portable across CPU architectures)`); + } +} + +async function checkQuickjs(bundleDir) { + const quickjsUrl = pathToFileURL(join(bundleDir, "node_modules", "quickjs-wasi", "dist", "index.js")).href; + const mod = await import(quickjsUrl); + if (typeof mod.QuickJS?.create !== "function") fail("quickjs-wasi does not export QuickJS.create"); + const wasmPath = join(bundleDir, "node_modules", "quickjs-wasi", "quickjs.wasm"); + // Mirror the real consumer (pi-codemode): resolve quickjs-wasi/quickjs.wasm + // through node resolution and pass the bytes to QuickJS.create. + const wasmBytes = readFileSync(wasmPath); + let vm; + try { + vm = await mod.QuickJS.create({ wasm: wasmBytes }); + if (!vm?.exports?.memory) fail("quickjs-wasm module failed to instantiate (no exported memory)"); + const versions = vm.versions; + if (!versions?.quickjs) fail("instantiated quickjs reports no engine version"); + ok(`quickjs-wasi WebAssembly instantiates from the bundle (quickjs ${versions.quickjs})`); + } finally { + vm?.dispose?.(); + } +} + +// --------------------------------------------------------------------------- +// 4. Scripted session driving the real write tool (isolated agent-worker recipe) +// --------------------------------------------------------------------------- + +const PROOF_TEXT = `pi-sdk bundle verification ${new Date().toISOString()}\n`; + +async function scriptedToolSession(bundleDir, modules, canary) { + const { sdk, ai, agentCore } = modules; + const tempDir = mkdtempSync(join(tmpdir(), "pi-sdk-verify-")); + cleanupFns.push(() => rmSync(tempDir, { recursive: true, force: true })); + const agentDir = join(tempDir, "agent"); + mkdirSync(agentDir, { recursive: true }); + // Project-scope discovery canaries (cwd), in addition to the HOME ones. + const projectAgentDir = join(tempDir, ".pi", "agent", "extensions"); + mkdirSync(projectAgentDir, { recursive: true }); + writeFileSync( + join(projectAgentDir, "canary-project-ext.mjs"), + [ + 'import { writeFileSync } from "node:fs";', + 'writeFileSync(process.env.PI_VERIFY_CANARY_DIR + "/project-extension-executed", "yes");', + "export default {};", + "", + ].join("\n"), + ); + writeFileSync(join(tempDir, "AGENTS.md"), "CANARY-CONTEXT-FILE must never be loaded.\n"); + + // Isolation recipe (mirrors portable/workspace/agent-worker.mjs): explicit + // in-memory everything, discovery disabled, no network, explicit prompt. + const settingsManager = sdk.SettingsManager.inMemory({ + retry: { enabled: false, maxRetries: 0, provider: { maxRetries: 0 } }, + compaction: { enabled: false }, + cacheWarming: "off", + }); + const resourceLoader = new sdk.DefaultResourceLoader({ + cwd: tempDir, + agentDir, + settingsManager, + noExtensions: true, + noSkills: true, + noPromptTemplates: true, + noThemes: true, + noContextFiles: true, + systemPrompt: SYSTEM_PROMPT, + }); + await resourceLoader.reload(); + + const authStorage = (await importFromBundle(bundleDir, "pi-coding-agent/dist/core/auth-storage.js")).AuthStorage.inMemory({ + faux: { type: "api_key", key: "faux-key" }, + }); + const { InMemoryCodingAgentModelsStore } = await importFromBundle(bundleDir, "pi-coding-agent/dist/core/models-store.js"); + const modelRuntime = await sdk.ModelRuntime.create({ + credentials: authStorage, + modelsPath: null, + allowModelNetwork: false, + refreshOnCreate: false, + modelsStore: new InMemoryCodingAgentModelsStore(), + }); + const fauxModel = { + id: "faux-1", + name: "Faux Model", + api: "anthropic-messages", + provider: "faux", + baseUrl: "http://127.0.0.1:9", + reasoning: false, + input: ["text"], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow: 128000, + maxTokens: 16384, + }; + modelRuntime.registerProvider("faux", { + baseUrl: fauxModel.baseUrl, + api: fauxModel.api, + models: [{ ...fauxModel }], + }); + + const helloPath = join(tempDir, "hello.txt"); + const responses = [ + { toolCalls: [{ name: "write", args: { path: helloPath, content: PROOF_TEXT } }] }, + { text: "wrote the file" }, + ]; + let callCount = 0; + const streamFn = () => { + const response = responses[callCount++ % responses.length]; + const message = { + role: "assistant", + content: response.toolCalls + ? [ + { + type: "toolCall", + id: `verify_${callCount}`, + name: response.toolCalls[0].name, + arguments: response.toolCalls[0].args, + }, + ] + : [{ type: "text", text: response.text }], + api: fauxModel.api, + provider: fauxModel.provider, + model: fauxModel.id, + usage: { input: 10, output: 5, cacheRead: 0, cacheWrite: 0, totalTokens: 15, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, + stopReason: response.toolCalls ? "toolUse" : "stop", + timestamp: Date.now(), + }; + const stream = ai.createAssistantMessageEventStream(); + queueMicrotask(() => { + stream.push({ type: "start", partial: { ...message, content: [], stopReason: "pending" } }); + message.content.forEach((block, index) => { + if (block.type === "text") { + stream.push({ type: "text_start", contentIndex: index, partial: message }); + stream.push({ type: "text_delta", contentIndex: index, delta: block.text, partial: message }); + stream.push({ type: "text_end", contentIndex: index, content: block.text, partial: message }); + } else { + stream.push({ type: "toolcall_start", contentIndex: index, partial: message }); + stream.push({ type: "toolcall_end", contentIndex: index, toolCall: block, partial: message }); + } + }); + stream.push({ type: "done", reason: message.stopReason, message }); + }); + return stream; + }; + + const tools = sdk.createCodingTools(tempDir); + const toolRecord = Object.fromEntries(tools.map((tool) => [tool.name, tool])); + if (!toolRecord.write) fail("createCodingTools did not provide a write tool"); + + const agent = new agentCore.Agent({ + getApiKey: async () => "faux-key", + initialState: { model: fauxModel, systemPrompt: SYSTEM_PROMPT, tools: [] }, + streamFn, + }); + + const events = []; + const session = new sdk.AgentSession({ + agent, + sessionManager: sdk.SessionManager.inMemory(tempDir), + settingsManager, + cwd: tempDir, + modelRuntime, + resourceLoader, + baseToolsOverride: toolRecord, + }); + session.subscribe((event) => events.push(event)); + + try { + const timeout = new Promise((_, reject) => { + const timer = setTimeout(() => reject(new Error("session timed out after 90s")), 90000); + timer.unref(); + }); + timeout.catch(() => {}); // the race winner may finish first; keep the rejection handled + await Promise.race([session.prompt("Write hello.txt exactly as instructed."), timeout]); + } finally { + session.dispose(); + } + + if (!existsSync(helloPath)) fail("scripted session finished but the write tool did not create hello.txt"); + const written = readFileSync(helloPath, "utf8"); + if (written !== PROOF_TEXT) fail(`write tool produced wrong content: ${JSON.stringify(written)}`); + const toolExecutions = events.filter((event) => event.type.startsWith("tool_")); + if (toolExecutions.length === 0) fail("session emitted no tool events; the scripted tool call was not executed"); + const writeExecuted = events.some((event) => event.type.startsWith("tool_") && JSON.stringify(event).includes('"write"')); + if (!writeExecuted) fail("no tool event references the write tool"); + ok(`scripted session executed the real write tool (${toolExecutions.length} tool events, hello.txt verified)`); + + // --- Canary assertions: discovery must not have executed anything -------- + const markerDir = canary.markerDir; + const markerNames = ["extension-executed", "project-extension-executed"]; + for (const marker of markerNames) { + if (existsSync(join(markerDir, marker))) fail(`discovery canary executed: ${marker} was created; isolation is broken`); + } + const extensions = resourceLoader.getExtensions(); + if (extensions?.extensions?.length > 0) fail(`discovery canary: extensions were loaded (${extensions.extensions.length}) despite noExtensions`); + const skills = resourceLoader.getSkills(); + if (skills?.skills?.length > 0) fail(`discovery canary: skills were loaded (${skills.skills.length}) despite noSkills`); + const prompts = resourceLoader.getPrompts(); + if (prompts?.prompts?.length > 0) fail(`discovery canary: prompt templates were loaded (${prompts.prompts.length}) despite noPromptTemplates`); + const themes = resourceLoader.getThemes(); + if (themes?.themes?.length > 0) fail(`discovery canary: themes were loaded (${themes.themes.length}) despite noThemes`); + const agentsFiles = resourceLoader.getAgentsFiles(); + if (agentsFiles?.agentsFiles?.length > 0) fail(`discovery canary: context files were loaded (${agentsFiles.agentsFiles.length}) despite noContextFiles`); + const systemPrompt = resourceLoader.getSystemPrompt() ?? ""; + if (systemPrompt.includes("CANARY")) fail("discovery canary: CANARY text leaked into the system prompt"); + if (!systemPrompt.includes("bundle verification assistant")) fail("explicit system prompt was not used"); + const credentialProviders = (await modelRuntime.listCredentials()).map((info) => info.providerId); + if (credentialProviders.includes("canary-provider")) fail("discovery canary: HOME auth.json was consulted by the model runtime"); + const modelIds = modelRuntime.getAllModels().map((model) => `${model.provider}/${model.id}`); + if (modelIds.some((id) => id.includes("canary"))) fail("discovery canary: HOME models.json was consulted by the model runtime"); + ok( + `isolation canaries clean: no extension/skill/prompt/theme/context-file execution, HOME auth.json+models.json untouched, explicit system prompt in effect (env reduced to allowlist + disposable HOME)`, + ); +} + +// --------------------------------------------------------------------------- + +async function main() { + const options = parseArgs(process.argv.slice(2)); + const bundleDir = options.bundle; + if (!existsSync(join(bundleDir, "node_modules", "@earendil-works", "pi-coding-agent"))) { + fail(`${bundleDir} does not contain node_modules/@earendil-works/pi-coding-agent`); + } + console.log(`verify-bundle: bundle: ${bundleDir}`); + console.log(`verify-bundle: host: ${process.platform}/${process.arch}, node ${process.version}`); + + // 0. Isolate the environment BEFORE importing any bundle code. + const canary = isolateEnvironment(); + console.log(`verify-bundle: env: reduced to allowlist [${ENV_ALLOWLIST.join(", ")}, HOME, TMPDIR, PI_OFFLINE] with disposable HOME`); + + // 1. Complete manifest check BEFORE any runtime import. + checkManifest(bundleDir); + + // 2..4 only run once integrity is established. + const modules = await probeSdk(bundleDir); + await checkArtifacts(bundleDir); + await checkQuickjs(bundleDir); + await scriptedToolSession(bundleDir, modules, canary); + ok("all bundle checks passed (scope: manifest integrity, SDK import, native/WASM presence + host-arch load, quickjs, write tool; NOT universal SDK completeness, NOT target-arch proof)"); + cleanup(); +} + +main().catch((error) => { + fail(error?.stack || String(error)); +}); diff --git a/portable/raspberry-pi/verify-workspace.py b/portable/raspberry-pi/verify-workspace.py new file mode 100644 index 0000000..31d8b23 --- /dev/null +++ b/portable/raspberry-pi/verify-workspace.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +"""Exercise the real terminal through the loopback HTTP API inside a guest.""" +import json +import time +import urllib.request + +base = "http://127.0.0.1:8765" + +def request(path, body=None, method=None, token=None): + headers = {"Content-Type": "application/json"} + if token: + headers["X-Workspace-Token"] = token + req = urllib.request.Request(base + path, + data=None if body is None else json.dumps(body).encode(), + headers=headers, method=method) + with urllib.request.urlopen(req, timeout=30) as response: + return json.load(response) + +status = request("/api/status") +token = status["csrfToken"] +terminal = request("/api/terminals", {}, token=token)["id"] +try: + request(f"/api/terminals/{terminal}/input", + {"input": "printf 'ARM_TERMINAL_'; uname -m\n"}, token=token) + output = "" + for _ in range(30): + output = request(f"/api/terminals/{terminal}/output?after=0")["output"] + if "ARM_TERMINAL_aarch64" in output: + break + time.sleep(1) + assert "ARM_TERMINAL_aarch64" in output, output + print(json.dumps({"terminal": "passed", "architecture": "aarch64", + "qualification": status.get("qualification")})) +finally: + request(f"/api/terminals/{terminal}", method="DELETE", token=token) diff --git a/portable/workspace/README.md b/portable/workspace/README.md new file mode 100644 index 0000000..7591179 --- /dev/null +++ b/portable/workspace/README.md @@ -0,0 +1,249 @@ +# Portable workspace prototype + +A new workspace presentation for portable packaging, independent of the existing +launcher and terminal UI. Terminal sessions are real Linux PTYs running as the +account that starts the server. The agent tab is backed by a real lifecycle +adapter around the vendored Pi SDK, driven by a scripted local model — not by +provider inference. Network and funding states remain fixtures. + +Requires Linux, Node 24, Python 3 and Bash. No npm installation or build is +needed for terminals and tests. Use an explicitly disposable directory: + +```sh +mkdir -p "$PWD/local-data/portable-demo" +node portable/workspace/server.mjs --workspace "$PWD/local-data/portable-demo" +``` + +Open `http://127.0.0.1:8765`. The server binds only to IPv4 loopback. When running +remotely, forward this port through SSH and open that same local address. Changing +the forwarding port requires using the same port on both ends because Host checks +are exact. `--port` selects another server port. + +The terminal is a plain text PTY view, not a full terminal emulator. Shell state +persists across commands; Ctrl-C can interrupt foreground work. It starts in the +selected directory with a minimal environment and no shell startup files. That +directory is **not a sandbox**: commands have the launching account's filesystem +and network authority. Use a disposable guest for isolation. This prototype does +not enforce Tor and does not perform inference, payment, or balance queries. + +Closing a terminal hangs up its shell and foreground job. Programs can deliberately +detach; closing the UI tab does not shut down the server. Stop the server when the +rehearsal ends. Terminal output is bounded in memory, never logged by the server. +Workspace files survive shutdown. No payment credentials should be used here. + +## Agent adapter (local development prototype) + +The agent tab is served by a bounded lifecycle adapter, not simulated text from +the server. Each run spawns a dedicated worker process +([`agent-worker.mjs`](agent-worker.mjs)) that embeds the vendored Pi SDK when it +is built and reports a clear unavailable state when it is not. The worker selects +explicitly isolated SDK state: + +- `SettingsManager.inMemory()` with automatic retries disabled + (`retry.enabled=false`, `maxRetries=0`, provider `maxRetries=0`) so a future + real provider integration never retries silently, plus compaction and cache + warming off. +- `SessionManager.inMemory(cwd)`: no conversation files. +- `ModelRuntime.create` with an in-memory `AuthStorage` credential store and + `modelsPath: null`, `allowModelNetwork=false` and `refreshOnCreate=false`: + no `auth.json` or `models.json` is ever read from the workspace or any home + directory. Per-run SDK state lives in a fresh mode-0700 temporary directory + outside the workspace, created per run and removed when the worker exits. +- `DefaultResourceLoader` with extensions, skills, prompt templates, themes and + context files all disabled and an explicit system prompt: nothing ambient is + admitted. +- Only the `read` and `write` tools are admitted, bound to the workspace cwd. + +The worker's model is a **scripted local transport** registered as a native +provider (`abalone-scripted/scripted-agent`): it issues one real SDK tool call +(`write`, or `read` in tests) and then a final text message. Tool execution +inside the SDK is real; the "model responses" are a local script. This is +explicitly **not provider inference**: no network request is made, no +credential is read, and no paid API is contacted. Real inference requires a +future provider broker and is out of scope for this slice. + +### Scripted-proof tool outcome policy + +Because the scripted transport plans exactly one tool call, the worker treats +the run's honesty as depending on that one call: the planned tool must be one +of the admitted `read`/`write` tools (anything else is rejected before the +run), and the worker captures the SDK's own `tool_execution_end` results. Any +failed tool execution produces an error run — a `tool_failed` event, a nonzero +worker exit and no `done` — and `done.wrote` is only ever set for a captured, +successful `write` execution (reads never claim a write). This strictness is a +property of the one-tool scripted proof transport, **not** a general rule: a +future genuine model agent may observe intermediate tool errors and recover, +and must not be failed permanently by them. + +### Lifecycle semantics + +- A worker-reported terminal event (`done`, `aborted`, …) is a protocol + outcome, not process termination. A run occupies capacity until its child + process is actually reaped; capacity starts being reserved synchronously at + start (before the per-run directory is created), so concurrent starts cannot + exceed the limit. Runs whose child has not exited are never pruned and can + always be stopped. +- A worker-reported `done` is **provisional evidence, never a public terminal + state and never a public `done` event**. On arrival the manager converts it + to a `worker_reported_done` evidence marker in the bounded stream and retains + the worker's payload privately; the `state` returned by the events endpoint + (and by the manager's `events()`/`list()`) stays nonterminal + (`running`/`starting`) and **no done-typed event is observable** until the + worker process is actually reaped. A consumer that stops polling on terminal + states — or one that renders events, like the UI, which prints its + completion message from a `done` event — can never mistake a done-then-hang + or done-then-nonzero worker for success. `state` is authoritative; the + `worker_reported_done` line is the worker's claim, kept as evidence with the + payload withheld. +- Terminal `done` requires the protocol completion **and** a clean process + exit (exit code 0, no kill signal), settled at reap: only then is the + **canonical `done` event — the privately retained worker payload — appended + to the stream, exactly once, as the final line**. A `done` followed by a + nonzero/unclean exit becomes `error` (a `worker_exit` event records why) + with no canonical `done`, and an exit 0 without any reported completion also + becomes `error`. Manager-initiated outcomes (`timeout`, `aborted`, `error`) + and a worker-reported `aborted` outrank a reported `done`; a late worker + event can never revive or replace a terminal state. +- Server shutdown and per-run stop await actual child reaping and the removal + of the per-run temporary directories this manager created; there is no + give-up timer that could abandon a live child. Once shutdown has begun, new + runs are refused (503 with `Agent manager is closing`). + +The worker environment is a fixed allowlist (`PATH`, `LANG`, `PI_OFFLINE=1`). +`HOME` is not forwarded and no ambient credentials or API keys are inherited, so +the SDK cannot discover ambient auth. `agentDir` is explicit, so nothing under +the operator's home directory is consulted. The workspace cwd is still not a +sandbox: the `write` tool writes with the launching account's authority. + +### Agent API + +Same security checks as the terminal API (exact Host, same-origin Origin and +Fetch Metadata, `X-Workspace-Token` + JSON content type for mutations, bounded +request bodies). + +- `GET /api/status` now includes `agent: { available, kind, sdkPath, detail }`, + where `kind` is `sdk` (built SDK), `fixture` (test transport) or `missing`. +- `POST /api/agents {prompt}` starts a run (`{id}`); prompt is 1..8192 bytes. + At most two runs are active (a live worker process counts, even one that + already reported `done`); beyond that the response is 409. If the SDK is not + built the response is 503 with the reason — never fake agent output. +- `GET /api/agents` lists runs (`{agents: [{id, state, alive}]}`) for reattach + after a page reload; `alive` reports process truth (whether the worker + process is still live), not the protocol claim. +- `GET /api/agents/:id/events?after=N` returns `{events, offset, truncated, + state}`. Events are the worker's JSONL records (ready, state, agent_event, + `worker_reported_done` evidence, the canonical done/aborted/error outcomes, + and `worker_exit`/`tool_failed` failure evidence). Poll with the previous + offset; the buffer is bounded to 256 KiB per run and reports truncation like + terminal output. **Neither a terminal `state` nor a done-typed event is + observable before the worker process is reaped**: a `worker_reported_done` + line is the worker's provisional claim with its payload withheld — keep + polling until `state` is terminal. The canonical `done` (carrying the + worker's payload, e.g. `lastText`/`wrote`) is appended exactly once, only + after the verified clean exit; every non-success settlement (`error`) + carries `worker_exit` evidence instead. +- `DELETE /api/agents/:id` cancels the run: the worker aborts the SDK session, + and an unresponsive worker is killed after a grace period. The response is + sent once the worker process is actually reaped and its per-run temp + directory is removed. A run also times out after 120 s (the timeout governs + the process even after a reported `done`). + +To run the adapter standalone against a built SDK (qualification; also covered +automatically by the gated smoke test below). The availability probe runs once +per server process, so building the SDK requires a server restart to be +detected. `agentDir` may be omitted: the worker then creates and removes its own +mode-0700 temporary directory outside the workspace: + +```sh +(cd vendor/pi && npm ci --no-audit --no-fund && bash -c "$(node -p 'require("./package.json").scripts["build:offline"].replace("cd ../coding-agent && npm run build", "cd ../coding-agent && npm run build:unbundled")')") +node portable/workspace/agent-worker.mjs --probe # expect {"type":"ready","kind":"sdk",...} +echo '{"prompt":"write the proof file","cwd":"'"$PWD/local-data/portable-demo"'"}' | node portable/workspace/agent-worker.mjs +``` + +`PORTABLE_PI_SDK=/abs/path/dist/index.js` overrides the SDK location. + +## API and adapter seam + +`GET /api/status` returns `csrfToken`, `workspace`, `terminalAvailable`, +`qualification` and `agent`. All mutations require `Content-Type: application/json` +and `X-Workspace-Token` with that token. Foreign Origins, cross-site Fetch +Metadata and unexpected Host Headers are refused; static serving is an explicit +file allowlist. + +- `POST /api/terminals {}` returns `{id}` (eight sessions maximum). +- `POST /api/terminals/:id/input {input}` queues raw terminal input. +- `GET /api/terminals/:id/output?after=0` returns `{output,offset,exited,truncated}`. + Poll with the previous offset; output includes terminal control characters. +- `DELETE /api/terminals/:id` closes the session. +- `GET /api/terminals` returns `{sessions: [{id, exited}]}` so a reloaded page + can reattach to surviving PTYs (via the output endpoint above) instead of + leaking new sessions until the eight-session cap is exhausted. The current UI + does not use it yet; the endpoint is the coordination point for that fix. + +A live provider adapter must keep the SDK's in-memory session manager, isolated +configuration and admitted tools, keep retries disabled, and route credentials +through a separate broker rather than the worker environment. Connection and +funding adapters must distinguish unknown balance, stale balance, payment +detection, confirmation, conversion and spendable credit. Do not upgrade fixture +labels to live claims without end-to-end evidence through the shipped network +boundary. + +## Checks + +```sh +node --test portable/workspace/server.test.mjs +node --test portable/workspace/agent.test.mjs +``` + +The first checks cross-origin/Host/token rejection, request limits, actual shell +state, workspace writes, foreground interruption, session closure and +static-file exposure. The second checks the agent lifecycle: event ordering, +mid-stream reattach offsets, cancellation and process reaping, timeout +escalation to kill, the unavailable-SDK refusal path (no fake output), the +allowlisted worker environment (an ambient secret canary is not inherited), +bounded event buffers, probe newline/buffer hardening (split status line, +unbounded noise), oversized-frame refusal with worker termination, monotonic +final states (no late-event revival), synchronous capacity reservation under +five concurrent starts against a limit of two, reported-done-is-not-termination +(a done-reporting live worker keeps capacity until reaped and is still killed +by its run timeout), repeated-poll proofs over the delayed-exit window that +assert the response **events** as well as the response **state** — no +done-typed event and no terminal state before the reap, on the manager +`events()` surface and over HTTP, for a done-then-hang worker (settled by its +run timeout), a delayed nonzero exit (settled as `error` with `worker_exit` +evidence) and a delayed clean exit (the canonical `done` appears exactly +once, appended at the verified reap and carrying the retained payload) — +plus a worker-reported `aborted` after `done` outranking the completion +claim, done-then-nonzero-exit and exit-0-without-done both settling as errors +with no canonical `done`, a racing start/close leaving no stray worker +processes or temp directories, agent-endpoint security checks, +and the terminal session listing for reload reattach. Its final tests are the gated actual-SDK checks: +the smoke check runs the real worker against the built vendored SDK and asserts +the real tool file contents and event order, that a poisoned workspace +`.pi-agent` (symlinked `auth.json`, canary `models.json`) is ignored in favor of +the in-memory credential store (verified via the runtime's own +`listCredentials`/model catalog, not mocked call options), and that the per-run +agentDir is a fresh 0700 temp directory outside the workspace that is removed +after the worker exits; the negative checks assert that a failed real SDK write +(to a directory path), a missing-file read and an invalid planned tool each +produce an error run with no `done` and no `wrote` claim, and that a successful +read completes without claiming a write. When `dist/` is absent those tests +report an explicit skip with the reason — they never masquerade as an +actual-SDK pass. + +### Qualification status and honest limits + +- The fixture transport ([`agent-fixture-worker.mjs`](agent-fixture-worker.mjs)) + covers the server/manager seam without needing a built SDK. +- Actual Pi SDK execution is qualified only by the gated smoke test (and the + standalone commands above), and only on machines where the vendored SDK is + built; the recorded run for this slice used SDK 1.1.0 built from + `vendor/pi` and passed. A checkout without `dist/` shows that test as + skipped, which is the honest result. +- The scripted model is not a model: responses are canned script turns. Only + the SDK tool execution path is real. +- No Tor, no sandbox, no privacy claim: this is local development tooling. The + worker writes with the launching account's authority inside the workspace. +- The repository offline integration gate and independent review remain required + before integration; these checks do not qualify ARM images, Tor or payment + behavior. diff --git a/portable/workspace/agent-fixture-worker.mjs b/portable/workspace/agent-fixture-worker.mjs new file mode 100644 index 0000000..a333878 --- /dev/null +++ b/portable/workspace/agent-fixture-worker.mjs @@ -0,0 +1,156 @@ +// Fixture transport worker for the portable workspace agent adapter. +// +// Speaks the same JSONL contract as agent-worker.mjs WITHOUT the Pi SDK, so +// lifecycle and security checks run without a built SDK. Every event is +// labeled kind:"fixture". This is not agent execution and not SDK qualification. +// +// Job line on stdin: {prompt, cwd, agentDir, script?} +// Fixture-only script knobs: reportEnv, slow, flood, hang, proofPath, +// proofContent, oversize, revive, doneThenHang, doneThenExit (with optional +// exitDelayMs), doneThenExit0 (done, optional delay, clean exit 0), +// doneThenAbort (done, then a worker-reported aborted, clean exit), +// exit0NoDone. +import fs from 'node:fs'; +import path from 'node:path'; + +console.log = (...args) => console.error(...args); +const emit = value => process.stdout.write(JSON.stringify(value) + '\n'); + +if (process.argv.includes('--probe')) { + emit({ type: 'ready', kind: 'fixture' }); + process.exit(0); +} + +let script = {}; +let cwd = null; +let prompt = ''; +try { + let data = ''; + const job = await new Promise((resolve, reject) => { + process.stdin.setEncoding('utf8'); + process.stdin.on('data', chunk => { + data += chunk; + if (Buffer.byteLength(data) > 65536) reject(new Error('Job too large')); + }); + process.stdin.on('end', () => resolve(JSON.parse(data || '{}'))); + process.stdin.on('error', reject); + }); + if (typeof job.prompt !== 'string' || !job.prompt.trim() || Buffer.byteLength(job.prompt) > 8192) throw new Error('invalid prompt'); + if (typeof job.cwd !== 'string' || !path.isAbsolute(job.cwd)) throw new Error('invalid cwd'); + cwd = job.cwd; + prompt = job.prompt; + script = job.script ?? {}; +} catch (error) { + emit({ type: 'fatal', message: error instanceof Error ? error.message : String(error) }); + process.exit(1); +} + +const delay = ms => new Promise(resolve => setTimeout(resolve, ms)); +const pace = script.slow ? 120 : 8; +let stopping = false; + +process.on('SIGTERM', () => { + stopping = true; +}); +process.on('SIGINT', () => { stopping = true; }); + +emit({ type: 'ready', kind: 'fixture' }); +if (script.reportEnv) { + emit({ type: 'worker_env', env: Object.keys(process.env).sort() }); +} +emit({ type: 'state', state: 'running' }); +emit({ type: 'agent_event', event: 'agent_start' }); +emit({ type: 'agent_event', event: 'turn_start' }); +emit({ type: 'agent_event', event: 'message_start', role: 'assistant' }); + +if (script.flood) { + // Large event volume to exercise the bounded event buffer. + for (let index = 0; index < 400 && !stopping; index++) { + emit({ type: 'agent_event', event: 'text_delta', delta: 'x'.repeat(120) }); + await delay(0); + } +} else { + emit({ + type: 'agent_event', event: 'tool_start', tool: 'write', + args: `{"path":"${script.proofPath ?? 'fixture-proof.txt'}"}`, + }); + if (!stopping) { + // Fixture stand-in for the SDK write tool: one bounded file inside cwd. + fs.writeFileSync( + path.join(cwd, script.proofPath ?? 'fixture-proof.txt'), + script.proofContent ?? `fixture\n${prompt.slice(0, 200)}\n`, + ); + } + emit({ type: 'agent_event', event: 'tool_end', tool: 'write', isError: false }); + if (script.oversize) { + // Malformed protocol mid-stream: one valid-JSON frame far beyond the + // frame limit; everything after it must be dropped by the manager. + emit({ type: 'agent_event', event: 'text_delta', delta: 'x'.repeat(300 * 1024) }); + } + emit({ type: 'agent_event', event: 'message_end', role: 'assistant', stopReason: 'toolUse' }); + emit({ type: 'agent_event', event: 'message_start', role: 'assistant' }); + for (const word of ['fixture ', 'agent ', 'reply']) { + if (stopping) break; + await delay(pace); + emit({ type: 'agent_event', event: 'text_delta', delta: word }); + } + emit({ type: 'agent_event', event: 'message_end', role: 'assistant', stopReason: 'stop' }); +} + +if (stopping) { + emit({ type: 'aborted', reason: 'cancelled' }); + process.exit(0); +} +if (script.hang) { + // Ignores SIGTERM to exercise the manager's escalation to SIGKILL. + process.removeAllListeners('SIGTERM'); + setInterval(() => {}, 60_000); // Keep the event loop alive. + await new Promise(() => {}); +} +if (script.doneThenHang) { + // A worker that reports protocol completion and then never exits: the + // manager must keep counting the live child, keep the record reachable + // for stop/close, and only settle the run once the process is reaped. + emit({ type: 'done', lastText: 'fixture reply' }); + process.removeAllListeners('SIGTERM'); + setInterval(() => {}, 60_000); + await new Promise(() => {}); +} +if (script.doneThenExit) { + // Reports done, then exits nonzero: reported completion is not success + // unless the process also exits cleanly. exitDelayMs optionally holds the + // done-reported-but-not-yet-exited gap open, so checks can observe (by + // repeated polling) that the public state stays nonterminal during it. + emit({ type: 'done', lastText: 'fixture reply' }); + if (script.exitDelayMs) await delay(Math.min(Number(script.exitDelayMs) || 0, 5000)); + process.exit(Number(script.doneThenExit) || 7); +} +if (script.doneThenExit0) { + // Reports done, then exits cleanly after an optional delay: the canonical + // done must appear exactly once, appended only after the verified reap and + // carrying the worker's retained payload. + emit({ type: 'done', lastText: 'fixture reply' }); + if (script.exitDelayMs) await delay(Math.min(Number(script.exitDelayMs) || 0, 5000)); + process.exit(0); +} +if (script.doneThenAbort) { + // Reports done, then retracts it with a worker-reported aborted outcome, + // then exits cleanly: aborted outranks the completion claim, so the run + // settles aborted and no canonical done is ever appended. + emit({ type: 'done', lastText: 'fixture reply' }); + emit({ type: 'aborted', reason: 'retracted' }); + process.exit(0); +} +if (script.exit0NoDone) { + // Exits 0 without ever reporting completion: must not become done. + process.exit(0); +} +emit({ type: 'agent_event', event: 'agent_end' }); +emit({ type: 'agent_event', event: 'agent_settled', aborted: false }); +emit({ type: 'done', lastText: 'fixture reply' }); +if (script.revive) { + // Late events after the terminal done: a run must not be revived. + emit({ type: 'state', state: 'running' }); + emit({ type: 'agent_event', event: 'text_delta', delta: 'late' }); +} +process.exit(0); diff --git a/portable/workspace/agent-manager.mjs b/portable/workspace/agent-manager.mjs new file mode 100644 index 0000000..fdd5f4a --- /dev/null +++ b/portable/workspace/agent-manager.mjs @@ -0,0 +1,428 @@ +// Agent lifecycle manager for the portable workspace server. +// +// Spawns one dedicated worker process per agent run (agent-worker.mjs, or an +// explicitly injected fixture worker for tests), with an allowlisted child +// environment: no inherited credentials, no ambient auth/resource discovery. +// Worker stdout is a JSONL event channel; buffers are bounded like terminal +// output. This is a local development prototype: the workspace directory is +// not a sandbox and no privacy or Tor enforcement is claimed. +// +// Lifecycle rules: +// - Capacity counts reserved starts (pendingStarts) plus records whose child +// process has not exited yet. Capacity is reserved synchronously — no await +// (including agentDir creation) may interleave between the check and the +// reservation — and a failed start rolls its reservation back. +// - A worker-reported terminal event (done/aborted/...) is a protocol +// outcome, not process termination. The record stays counted and is never +// pruned until the child is actually reaped; stop()/close() can still kill +// a worker that reported done and then hung. +// - A worker-reported "done" is provisional evidence, never a public +// terminal state and never a public "done" event. It is converted on +// arrival to a worker_reported_done evidence marker in the bounded stream, +// while the worker's payload is retained privately on the record. The +// state reported by events()/list() stays nonterminal (running/starting) +// and no done-typed event is observable until the process is actually +// reaped, so neither a state-polling nor an event-rendering consumer (the +// UI prints its completion message from a done event) can mistake a +// done-then-hang or done-then-nonzero worker for success. +// - Terminal "done" requires the protocol completion AND a clean process +// exit (code 0, no signal), settled at reap: only then is the canonical +// done event — the privately retained worker payload — appended to the +// stream. done followed by a nonzero/unclean exit becomes error with no +// canonical done; exit 0 with no reported completion becomes error. +// Manager-initiated outcomes (timeout, aborted, error) take precedence over +// a worker-reported done but never over each other; a late worker event can +// never revive or replace a terminal state. +// - close() rejects new starts, waits for in-flight starts to settle, stops +// every record and awaits both child reaping and removal of the per-run +// agentDirs this manager created before resolving. +import { spawn } from 'node:child_process'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import { fileURLToPath } from 'node:url'; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const defaultWorkerPath = path.join(here, 'agent-worker.mjs'); + +const TERMINAL_STATES = new Set(['done', 'aborted', 'error', 'timeout', 'killed', 'unavailable']); +// Manager-initiated outcomes outrank a worker-reported done (protocol +// completion is not termination) and never demote each other. 'done' itself +// is never set from a worker event: it is settled only at reap, when the +// protocol completion is confirmed by a clean exit. +const STATE_RANK = { starting: 0, running: 1, done: 2, unavailable: 3, aborted: 4, timeout: 5, killed: 6, error: 7 }; + +export function createAgentManager(options = {}) { + const { + workspace, + workerPath = defaultWorkerPath, + maxAgents = 2, + maxRecords = 16, + eventLimit = 256 * 1024, + runTimeoutMs = 120_000, + killGraceMs = 3_000, + probeTimeoutMs = 15_000, + probeLineLimit = 64 * 1024, + maxFrameBytes = 64 * 1024, + } = options; + if (!workspace || !path.isAbsolute(workspace)) throw new Error('An explicit absolute workspace is required'); + + const records = new Map(); + let probe = null; + let closing = false; + // Capacity reservations for starts that have not spawned their child yet; + // guards the await window (probe, agentDir creation) between the capacity + // check and the record becoming visible with a live child. + let pendingStarts = 0; + // Every start() attempt not yet settled, so close() can wait for starts + // that were already in flight when shutdown began. + const inFlightStarts = new Set(); + // Outstanding removals of per-run agentDirs created by this manager. + const dirCleanups = new Set(); + + const agentClosingError = () => { + const error = new Error('Agent manager is closing; no new runs are accepted'); + error.code = 'AGENT_CLOSING'; + return error; + }; + + // Removes only the exact per-run directory this manager created with + // mkdtemp, and tracks the removal so shutdown can await it. + const removeAgentDir = dir => { + const done = fs.rm(dir, { recursive: true, force: true }).catch(() => {}); + dirCleanups.add(done); + done.then(() => dirCleanups.delete(done), () => dirCleanups.delete(done)); + return done; + }; + + // Fixed allowlist: PATH for tool helpers, a locale, and PI_OFFLINE to keep + // the SDK from any model-catalog network refresh. HOME and every ambient + // credential variable stay out of the worker. + const childEnv = () => ({ + ...(process.env.PATH ? { PATH: process.env.PATH } : {}), + LANG: 'C.UTF-8', + PI_OFFLINE: '1', + }); + + // Waits for a complete newline-terminated line: a JSON document split + // across writes is not resolved early, and non-newline noise past + // probeLineLimit kills the probe instead of buffering without bound. + const firstJsonLine = child => new Promise((resolve, reject) => { + let buffer = ''; + const timer = setTimeout(() => { child.kill('SIGKILL'); reject(new Error('probe timed out')); }, probeTimeoutMs); + const fail = error => { clearTimeout(timer); reject(error); }; + child.once('error', fail); + child.stdout.setEncoding('utf8'); + child.stdout.on('data', chunk => { + buffer += chunk; + const newline = buffer.indexOf('\n'); + if (newline < 0) { + if (buffer.length > probeLineLimit) { + child.kill('SIGKILL'); + fail(new Error(`probe status line exceeded ${probeLineLimit} bytes`)); + } + return; + } + clearTimeout(timer); + try { resolve(JSON.parse(buffer.slice(0, newline))); } catch { fail(new Error('probe produced no JSON status line')); } + }); + child.once('close', () => { clearTimeout(timer); reject(new Error('probe exited before reporting')); }); + }); + + const probeWorker = () => { + if (!probe) { + probe = (async () => { + try { + const child = spawn(process.execPath, [workerPath, '--probe'], { stdio: ['ignore', 'pipe', 'ignore'], env: childEnv() }); + const status = await firstJsonLine(child); + child.kill('SIGKILL'); + if (status.type === 'ready') { + return { available: true, kind: status.kind ?? 'unknown', sdkPath: status.sdkPath ?? workerPath, detail: status.kind === 'sdk' ? `Pi SDK ${status.sdkVersion ?? 'unknown version'}` : 'fixture transport' }; + } + if (status.type === 'unavailable') { + return { available: false, kind: 'missing', sdkPath: status.sdkPath ?? workerPath, detail: status.reason ?? 'Pi SDK unavailable' }; + } + return { available: false, kind: 'unknown', sdkPath: workerPath, detail: `probe reported ${JSON.stringify(status.type)}` }; + } catch (error) { + return { available: false, kind: 'missing', sdkPath: workerPath, detail: error instanceof Error ? error.message : String(error) }; + } + })(); + } + return probe; + }; + + // Live occupancy: reserved starts plus records whose child process has not + // exited. A worker that reported done but still holds its process keeps + // occupying a slot until it is actually reaped. + const liveCount = () => pendingStarts + [...records.values()].filter(record => !record.reaped).length; + + // Prunes only records that are both protocol-terminal and reaped: a + // done-reporting but still-live worker must stay reachable so stop() and + // close() can kill it. + const prune = () => { + const finished = [...records.entries()].filter(([, record]) => TERMINAL_STATES.has(record.state) && record.reaped); + while (records.size > maxRecords && finished.length) { + const [id] = finished.shift(); + records.delete(id); + } + }; + + const append = (record, event) => { + const line = JSON.stringify(event); + const bytes = Buffer.byteLength(line) + 1; + record.lines.push({ event, bytes }); + record.bytes += bytes; + while (record.bytes > eventLimit && record.lines.length > 1) { + const dropped = record.lines.shift(); + record.bytes -= dropped.bytes; + record.base += dropped.bytes; + record.truncated = true; + } + }; + + const reap = record => { + if (record.timer) clearTimeout(record.timer); + if (record.killTimer) clearTimeout(record.killTimer); + }; + + const terminate = (record, state) => { + if (record.child.exitCode !== null || record.child.signalCode !== null) return; + record.enforce(state); + try { record.child.kill('SIGTERM'); } catch { /* already gone */ } + if (record.killTimer) clearTimeout(record.killTimer); + record.killTimer = setTimeout(() => { try { record.child.kill('SIGKILL'); } catch { /* already gone */ } }, killGraceMs); + }; + + const startRun = async ({ prompt, script } = {}) => { + const status = await probeWorker(); + if (!status.available) { + const error = new Error(status.detail || 'Agent SDK unavailable'); + error.code = 'AGENT_UNAVAILABLE'; + throw error; + } + if (closing) throw agentClosingError(); + if (liveCount() >= maxAgents) { + const error = new Error('Agent capacity reached; stop a running agent first'); + error.code = 'AGENT_CAPACITY'; + throw error; + } + // Reserve capacity synchronously: no await may run between the check + // above and this increment. Holds through agentDir creation and spawn. + pendingStarts += 1; + let agentDir = null; + try { + // Per-run agent directory: fresh, mode 0700 (mkdtemp), outside the + // workspace. The worker never sees a path inside the workspace for + // SDK state, and only this exact created directory is removed. + agentDir = await fs.mkdtemp(path.join(os.tmpdir(), 'pi-agent-')); + if (closing) throw agentClosingError(); + const id = crypto.randomUUID(); + const record = { + id, state: 'starting', child: null, reaped: false, lines: [], bytes: 0, base: 0, truncated: false, + timer: null, killTimer: null, startedAt: Date.now(), agentDir, malformed: false, cleaned: false, + closed: null, cleanupDone: null, completion: false, completionPayload: null, + }; + // Worker-protocol transitions are strictly monotonic: once + // terminal, no late worker event (a stray "running" after done, a + // done after timeout) can revive or replace the terminal state. A + // worker "done" does not transition at all: it only records the + // provisional completion; the public state stays nonterminal until + // the close handler settles it at reap. + record.transition = state => { + if (!TERMINAL_STATES.has(record.state)) record.state = state; + }; + // Manager-initiated outcomes (timeout/aborted/error) may supersede + // a worker-reported done, never each other, never downward. + record.enforce = state => { + if ((STATE_RANK[state] ?? 0) > (STATE_RANK[record.state] ?? 0)) record.state = state; + }; + let settleClosed; + record.closed = new Promise(resolve => { settleClosed = resolve; }); + const cleanup = () => { + if (record.cleaned) return record.cleanupDone; + record.cleaned = true; + record.cleanupDone = removeAgentDir(agentDir); + return record.cleanupDone; + }; + + // Spawn before the record becomes visible: no consumer of + // records/list() ever observes a child:null record. + const child = spawn(process.execPath, [workerPath], { stdio: ['pipe', 'pipe', 'ignore'], env: childEnv() }); + record.child = child; + let buffer = ''; + const handleLine = line => { + if (!line.trim() || record.malformed) return; + let event; + try { event = JSON.parse(line); } catch { + append(record, { type: 'worker_log', line: line.slice(0, 500) }); + return; + } + // A worker "done" is converted, never published as-is: the public + // stream gains only a worker_reported_done evidence marker, and + // the payload is retained privately on the record. The canonical + // done (this payload) is appended only once the reap verifies a + // clean exit, so no done-typed event is observable before that — + // a consumer that renders events, like the UI, cannot print its + // completion message from a provisional claim. + if (event.type === 'done') { + record.completion = true; + record.completionPayload = event; + append(record, { type: 'worker_reported_done' }); + return; + } + append(record, event); + switch (event.type) { + case 'state': + if (event.state === 'running') record.transition('running'); + break; + case 'aborted': record.transition('aborted'); break; + case 'unavailable': record.transition('unavailable'); break; + default: break; + } + }; + // A single frame larger than maxFrameBytes is refused before parsing or + // retention: nothing oversized is buffered into the event record, and a + // worker emitting one is terminated as malformed. + const refuseFrame = bytes => { + if (record.malformed) return; + record.malformed = true; + append(record, { type: 'frame_too_large', bytes, limit: maxFrameBytes }); + record.enforce('error'); + terminate(record, 'error'); + }; + child.stdout.setEncoding('utf8'); + child.stdout.on('data', chunk => { + if (record.malformed) return; + buffer += chunk; + let index; + while ((index = buffer.indexOf('\n')) >= 0) { + const line = buffer.slice(0, index); + buffer = buffer.slice(index + 1); + if (line.length > maxFrameBytes) return refuseFrame(line.length); + handleLine(line); + } + if (buffer.length > maxFrameBytes) refuseFrame(buffer.length); + }); + child.stdin.on('error', () => {}); + child.on('error', error => { + record.enforce('error'); + append(record, { type: 'error', message: error.message }); + // Spawn errors never produce a close event: the record is + // reaped here so capacity, prune and stop() see the truth. + record.reaped = true; + cleanup(); + settleClosed(); + }); + child.on('close', (code, signal) => { + reap(record); + record.reaped = true; + if (!record.malformed && buffer.trim()) handleLine(buffer); + // Settlement happens only at reap. If a terminal state is already + // set (manager-initiated timeout/abort/error, or a worker-reported + // aborted/unavailable) it keeps precedence and is left untouched; + // otherwise the run settles by combining the provisional protocol + // completion with process exit truth. + if (!TERMINAL_STATES.has(record.state)) { + if (record.completion && code === 0 && signal === null) { + // Protocol completion confirmed by a clean exit: this, and + // only this, is the terminal success, and only now is the + // canonical done — the privately retained worker payload — + // appended to the public stream. + record.state = 'done'; + append(record, record.completionPayload); + } else if (record.completion) { + // Reported completion but the process did not exit cleanly + // (nonzero code or kill signal): not a success. + record.state = 'error'; + append(record, { type: 'worker_exit', code, signal: signal ?? null, detail: 'worker reported done but exited uncleanly' }); + } else { + // No protocol completion was ever reported. + record.state = 'error'; + append(record, { type: 'worker_exit', code, signal: signal ?? null, detail: 'worker exited without reporting completion' }); + } + } + prune(); + cleanup(); + settleClosed(); + }); + records.set(id, record); + prune(); + pendingStarts -= 1; + record.timer = setTimeout(() => terminate(record, 'timeout'), runTimeoutMs); + try { + child.stdin.write(JSON.stringify({ prompt, cwd: workspace, agentDir, script: script ?? null }) + '\n'); + child.stdin.end(); + } catch { /* child vanished instantly; its close handler owns the record */ } + return { id }; + } catch (error) { + pendingStarts -= 1; + // The record was never published (spawn happens before + // records.set), so only the reservation and directory roll back. + if (agentDir) await removeAgentDir(agentDir); + throw error; + } + }; + + const start = options => { + if (closing) return Promise.reject(agentClosingError()); + const attempt = startRun(options); + inFlightStarts.add(attempt); + return attempt.finally(() => inFlightStarts.delete(attempt)); + }; + + const get = id => records.get(id) ?? null; + + const events = (id, after = 0) => { + const record = records.get(id); + if (!record) return null; + if (!Number.isSafeInteger(after) || after < 0) throw new Error('Invalid offset'); + const out = []; + let offset = record.base; + for (const line of record.lines) { + const start = offset; + offset += line.bytes; + if (start >= after) out.push(line.event); + } + return { events: out, offset, truncated: after < record.base, state: record.state }; + }; + + // Cancels a run and resolves only after the child is actually reaped and + // this run's agentDir removal has settled. There is no give-up timeout + // that could abandon a live child: terminate() escalates SIGTERM->SIGKILL. + const stop = id => { + const record = records.get(id); + if (!record) return Promise.resolve(false); + return (async () => { + terminate(record, 'aborted'); + // record.closed resolves only after the child's close/error handler + // has run, and that handler always starts the agentDir cleanup first. + await record.closed; + await record.cleanupDone; + records.delete(id); + return true; + })(); + }; + + const list = () => [...records.values()].map(record => ({ + id: record.id, + state: record.state, + // Process truth, not protocol outcome: a worker that reported done but + // still holds its process is listed alive (and still occupies capacity). + alive: !record.reaped, + })); + + const close = async () => { + closing = true; + // Starts already in flight when shutdown began must settle (they + // re-check closing after each await and roll back); no new start can + // pass the entry check. + while (inFlightStarts.size > 0) await Promise.allSettled([...inFlightStarts]); + await Promise.all([...records.keys()].map(id => stop(id))); + // Shutdown is done only once every created per-run agentDir removal + // (including those of already-pruned records) has settled. + while (dirCleanups.size > 0) await Promise.allSettled([...dirCleanups]); + }; + + return { status: probeWorker, start, get, events, stop, list, close }; +} diff --git a/portable/workspace/agent-worker.mjs b/portable/workspace/agent-worker.mjs new file mode 100644 index 0000000..8ee4526 --- /dev/null +++ b/portable/workspace/agent-worker.mjs @@ -0,0 +1,400 @@ +// Dedicated Pi SDK worker process for the portable workspace agent adapter. +// +// Contract: one JSON job line on stdin ({prompt, cwd, agentDir?, script?}), +// JSONL progress events on stdout, console noise rerouted to stderr. +// Exit codes: 0 done/aborted, 1 error, 2 usage, 3 SDK unavailable. +// agentDir is optional: when omitted the worker creates a mode 0700 temporary +// directory outside the workspace for the run and removes it before exiting. +// When provided (manager path), the caller owns creation and cleanup. +// +// The worker embeds the vendored Pi SDK when it is built and drives it with a +// SCRIPTED local model: no provider inference, no network calls, no paid API. +// Tool execution inside the SDK is real (the admitted `read`/`write` tools run +// against the explicit cwd). This is local development tooling: cwd is not a +// sandbox, and nothing here enforces privacy or Tor routing. +import path from 'node:path'; +import fs from 'node:fs'; +import os from 'node:os'; +import { fileURLToPath, pathToFileURL } from 'node:url'; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const repoRoot = path.resolve(here, '..', '..'); +const defaultSdkPath = path.join(repoRoot, 'vendor', 'pi', 'packages', 'coding-agent', 'dist', 'index.js'); +const sdkPath = process.env.PORTABLE_PI_SDK || defaultSdkPath; + +// Keep stdout a pure JSONL channel. +console.log = (...args) => console.error(...args); +console.info = (...args) => console.error(...args); + +const emit = value => process.stdout.write(JSON.stringify(value) + '\n'); +const PROBE = process.argv.includes('--probe'); + +function unavailable(reason) { + emit({ type: 'unavailable', reason, sdkPath }); + process.exit(3); +} + +async function importSdk() { + try { + return { sdk: await import(pathToFileURL(sdkPath).href) }; + } catch (error) { + return { error: error instanceof Error ? error.message : String(error) }; + } +} + +if (PROBE) { + const { sdk, error } = await importSdk(); + if (error) unavailable(`Pi SDK import failed at ${sdkPath}: ${error}`); + emit({ type: 'ready', kind: 'sdk', sdkVersion: sdk.VERSION, sdkPath }); + process.exit(0); +} + +// --------------------------------------------------------------------------- +// Scripted model transport (no pi-ai imports needed; same event contract). +// --------------------------------------------------------------------------- + +const SCRIPTED_PROVIDER = 'abalone-scripted'; +const SCRIPTED_MODEL_ID = 'scripted-agent'; +const SCRIPTED_API = 'abalone-scripted'; + +const zeroUsage = () => ({ + input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, +}); + +const scriptedModel = { + id: SCRIPTED_MODEL_ID, + name: 'Portable workspace scripted local agent (no inference)', + api: SCRIPTED_API, + provider: SCRIPTED_PROVIDER, + baseUrl: 'urn:abalone:scripted', // Never fetched; the scripted transport performs no I/O. + input: ['text'], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + reasoning: false, + contextWindow: 128000, + maxTokens: 4096, +}; + +class SimpleEventStream { + constructor() { + this.queue = []; + this.waiters = []; + this.finished = false; + this.resultPromise = new Promise(resolve => { this.resolveResult = resolve; }); + } + push(event) { + if (this.finished) return; + if (event.type === 'done') { this.finished = true; this.resolveResult(event.message); } + if (event.type === 'error') { this.finished = true; this.resolveResult(event.error); } + const waiter = this.waiters.shift(); + if (waiter) waiter({ value: event, done: false }); + else this.queue.push(event); + } + async *[Symbol.asyncIterator]() { + for (;;) { + const event = this.queue.shift(); + if (event) { yield event; continue; } + if (this.finished) return; + const result = await new Promise(resolve => this.waiters.push(resolve)); + if (result.done) return; + yield result.value; + } + } + result() { return this.resultPromise; } +} + +const tick = ms => new Promise(resolve => setTimeout(resolve, ms)); + +/** Scripted assistant turn: a tool call first, plain text once results return. */ +function scriptedStream(model, context, options) { + const messages = context.messages ?? []; + const hasToolResult = messages.some(m => m.role === 'toolResult'); + const stream = new SimpleEventStream(); + const signal = options?.signal; + (async () => { + const base = { + role: 'assistant', content: [], api: model.api, provider: model.provider, + model: model.id, usage: zeroUsage(), stopReason: 'pending', timestamp: Date.now(), + }; + const snap = () => ({ ...base, content: base.content.map(block => ({ ...block })) }); + const finishWith = (final, reason) => { + final.stopReason = reason; + stream.push({ type: 'done', reason, message: final }); + }; + const abort = () => { + const aborted = { ...snap(), stopReason: 'aborted' }; + stream.push({ type: 'error', reason: 'aborted', error: aborted }); + }; + try { + stream.push({ type: 'start', partial: snap() }); + if (signal?.aborted) return abort(); + if (!hasToolResult) { + const toolCall = { + type: 'toolCall', id: `scripted-${Date.now()}`, name: script.toolName, + arguments: { path: script.path, content: script.content }, + }; + base.content.push(toolCall); + stream.push({ type: 'toolcall_start', contentIndex: 0, partial: snap() }); + await tick(5); + if (signal?.aborted) return abort(); + stream.push({ type: 'toolcall_end', contentIndex: 0, toolCall: { ...toolCall }, partial: snap() }); + return finishWith(snap(), 'toolUse'); + } + const finalText = script.finalText; + base.content.push({ type: 'text', text: '' }); + stream.push({ type: 'text_start', contentIndex: 0, partial: snap() }); + for (const piece of finalText.match(/[\s\S]{1,32}/g) ?? []) { + await tick(5); + if (signal?.aborted) return abort(); + base.content[0].text += piece; + stream.push({ type: 'text_delta', contentIndex: 0, delta: piece, partial: snap() }); + } + stream.push({ type: 'text_end', contentIndex: 0, content: finalText, partial: snap() }); + finishWith(snap(), 'stop'); + } catch (error) { + console.error('scripted stream failed:', error); + abort(); + } + })(); + return stream; +} + +const scriptedProvider = { + id: SCRIPTED_PROVIDER, + name: 'Portable workspace scripted transport (no network)', + auth: { + apiKey: { + name: 'Scripted local auth (never requests anything)', + resolve: async () => ({ auth: {}, source: 'scripted-local' }), + }, + }, + getModels: () => [scriptedModel], + getAllModels: () => [scriptedModel], + stream: (model, context, options) => scriptedStream(model, context, options), + streamSimple: (model, context, options) => scriptedStream(model, context, options), +}; + +// --------------------------------------------------------------------------- +// Job handling +// --------------------------------------------------------------------------- + +let script = null; +// Set when this worker created its own per-run agent directory; only that +// directory is ever removed by this process. +let ownedAgentDir = null; +const cleanupAgentDir = () => { + if (!ownedAgentDir) return; + try { fs.rmSync(ownedAgentDir, { recursive: true, force: true }); } catch { /* best effort */ } +}; + +function readJob() { + return new Promise((resolve, reject) => { + let data = ''; + process.stdin.setEncoding('utf8'); + process.stdin.on('data', chunk => { + data += chunk; + if (Buffer.byteLength(data) > 65536) reject(new Error('Job too large')); + }); + process.stdin.on('end', () => resolve(JSON.parse(data || '{}'))); + process.stdin.on('error', reject); + }); +} + +function validateJob(job) { + if (typeof job.prompt !== 'string' || !job.prompt.trim() || Buffer.byteLength(job.prompt) > 8192) { + throw new Error('prompt must be a non-empty string of at most 8192 bytes'); + } + if (typeof job.cwd !== 'string' || !path.isAbsolute(job.cwd)) throw new Error('cwd must be absolute'); + if (job.agentDir !== undefined && (typeof job.agentDir !== 'string' || !path.isAbsolute(job.agentDir))) { + throw new Error('agentDir must be absolute when provided'); + } + script = { + toolName: 'write', + path: 'agent-run.txt', + content: `scripted-agent\n${job.prompt.slice(0, 200)}\n`, + finalText: 'Scripted agent finished: it exercised the real SDK write tool locally. This was not model inference.', + }; + const raw = job.script ?? {}; + for (const key of Object.keys(raw)) { + if (!['toolName', 'path', 'content', 'finalText'].includes(key)) throw new Error(`Unknown script field: ${key}`); + if (typeof raw[key] !== 'string' || Buffer.byteLength(raw[key]) > 2000) { + throw new Error(`script.${key} must be a string of at most 2000 bytes`); + } + script[key] = raw[key]; + } + // The scripted proof transport plans exactly one tool call, and it must be + // one of the two admitted tools: an unknown planned tool is rejected here + // instead of ever being reported as a successful run. (Defense in depth: + // the SDK itself also errors unknown tool calls, and the captured-outcome + // check after the run refuses success on any failed tool execution.) + if (!['read', 'write'].includes(script.toolName)) { + throw new Error(`script.toolName must be "read" or "write" for the scripted proof transport (got ${JSON.stringify(script.toolName)})`); + } +} + +const SYSTEM_PROMPT = [ + 'You are the scripted agent of the portable workspace prototype.', + 'Responses are produced by a local script, not a model. Use only the admitted tools.', +].join('\n'); + +let session = null; +let settled = { done: false, aborted: false }; +// Actual tool executions the SDK performed this run, as the SDK reported +// them: the planned scripted call is only believed when a real non-error +// execution of that tool was observed. +const toolOutcomes = []; + +const forward = event => { + switch (event.type) { + case 'agent_start': return emit({ type: 'agent_event', event: 'agent_start' }); + case 'turn_start': return emit({ type: 'agent_event', event: 'turn_start' }); + case 'turn_end': return emit({ type: 'agent_event', event: 'turn_end' }); + case 'agent_end': return emit({ type: 'agent_event', event: 'agent_end' }); + case 'agent_settled': settled = { done: true, aborted: event.aborted }; return emit({ type: 'agent_event', event: 'agent_settled', aborted: event.aborted }); + case 'message_start': return emit({ type: 'agent_event', event: 'message_start', role: event.message?.role }); + case 'message_update': + if (event.assistantMessageEvent?.type === 'text_delta') { + return emit({ type: 'agent_event', event: 'text_delta', delta: String(event.assistantMessageEvent.delta).slice(0, 500) }); + } + return; + case 'message_end': { + const message = event.message ?? {}; + return emit({ type: 'agent_event', event: 'message_end', role: message.role, stopReason: message.stopReason }); + } + case 'tool_execution_start': + return emit({ + type: 'agent_event', event: 'tool_start', tool: event.toolName, + args: String(JSON.stringify(event.args ?? null)).slice(0, 400), + }); + case 'tool_execution_end': + toolOutcomes.push({ tool: event.toolName, isError: event.isError === true }); + return emit({ type: 'agent_event', event: 'tool_end', tool: event.toolName, isError: event.isError === true }); + case 'auto_retry_start': + case 'auto_retry_end': + return emit({ type: 'agent_event', event: event.type, ...(event.type === 'auto_retry_start' ? { attempt: event.attempt } : {}) }); + default: return; + } +}; + +let stopping = false; +async function stop(reason) { + if (stopping) return; + stopping = true; + try { if (session) await session.abort(); } catch (error) { console.error('abort failed:', error); } + emit({ type: 'aborted', reason }); + cleanupAgentDir(); + process.exit(0); +} +process.on('SIGTERM', () => void stop('cancelled')); +process.on('SIGINT', () => void stop('interrupted')); + +async function importAuthStorage() { + const url = pathToFileURL(path.join(path.dirname(sdkPath), 'core', 'auth-storage.js')).href; + try { + return { AuthStorage: (await import(url)).AuthStorage }; + } catch (error) { + return { error: `AuthStorage import failed next to ${sdkPath}: ${error instanceof Error ? error.message : String(error)}` }; + } +} + +try { + const job = await readJob(); + validateJob(job); + const { sdk, error } = await importSdk(); + if (error) unavailable(`Pi SDK import failed at ${sdkPath}: ${error}`); + const { AuthStorage, error: authError } = await importAuthStorage(); + if (authError) throw new Error(authError); + + let agentDir = job.agentDir ?? null; + if (agentDir) fs.mkdirSync(agentDir, { recursive: true, mode: 0o700 }); + else agentDir = ownedAgentDir = fs.mkdtempSync(path.join(os.tmpdir(), 'pi-agent-')); + emit({ type: 'ready', kind: 'sdk', sdkVersion: sdk.VERSION, sdkPath, model: `${SCRIPTED_PROVIDER}/${SCRIPTED_MODEL_ID}` }); + + // Explicit in-memory and isolated state; nothing discovers HOME configuration. + const settingsManager = sdk.SettingsManager.inMemory({ + // Automatic retries stay off for any future real provider integration. + retry: { enabled: false, maxRetries: 0, provider: { maxRetries: 0 } }, + compaction: { enabled: false }, + cacheWarming: 'off', + }); + const sessionManager = sdk.SessionManager.inMemory(job.cwd); + // Credentials come from a real in-memory AuthStorage (imported from the SDK + // dist core because it is not a public index export); no auth.json or + // models.json path is configured anywhere, so nothing under the workspace + // or a home directory is consulted for auth or model configuration. + const modelRuntime = await sdk.ModelRuntime.create({ + credentials: AuthStorage.inMemory({ [SCRIPTED_PROVIDER]: { type: 'api_key', key: 'portable-scripted-local' } }), + modelsPath: null, + allowModelNetwork: false, + refreshOnCreate: false, + }); + modelRuntime.registerNativeProvider(scriptedProvider); + const resourceLoader = new sdk.DefaultResourceLoader({ + cwd: job.cwd, + agentDir, + settingsManager, + noExtensions: true, + noSkills: true, + noPromptTemplates: true, + noThemes: true, + noContextFiles: true, + systemPrompt: SYSTEM_PROMPT, + }); + await resourceLoader.reload(); + const model = modelRuntime.getModel(SCRIPTED_PROVIDER, SCRIPTED_MODEL_ID) ?? scriptedModel; + const created = await sdk.createAgentSession({ + cwd: job.cwd, + agentDir, + model, + thinkingLevel: 'off', + modelRuntime, + resourceLoader, + sessionManager, + settingsManager, + tools: ['read', 'write'], + }); + session = created.session; + session.subscribe(forward); + emit({ type: 'state', state: 'running' }); + await session.prompt(job.prompt); + if (!stopping) { + if (settled.aborted) { + emit({ type: 'aborted', reason: 'run' }); + } else { + // Actual-SDK evidence for isolation checks: what the runtime's own + // credential store and model catalog report after the run. + try { + const credentialProviders = (await modelRuntime.listCredentials()).map(info => info.providerId); + const models = modelRuntime.getAllModels().map(model => `${model.provider}/${model.id}`).slice(0, 300); + emit({ type: 'sdk_diag', credentialProviders, models }); + } catch (diagError) { + emit({ type: 'sdk_diag', error: String(diagError instanceof Error ? diagError.message : diagError).slice(0, 300) }); + } + // Scripted-proof outcome policy: the script plans exactly ONE tool + // call, so any failed (or, if tool admission ever diverges, unknown) + // tool execution means the planned proof did not happen — report an + // error run (no done, nonzero exit) instead of success. This + // strictness is a property of the one-tool scripted transport, NOT a + // general rule: a future genuine model agent may observe tool errors + // mid-run and recover, and must not be failed permanently by them. + const failed = toolOutcomes.find(outcome => outcome.isError); + if (failed) { + emit({ type: 'tool_failed', tool: failed.tool, plannedTool: script.toolName }); + session.dispose(); + cleanupAgentDir(); + process.exit(1); + } + // Only a captured, successful `write` execution justifies a wrote + // claim; reads never claim a write. + const writeSucceeded = toolOutcomes.some(outcome => outcome.tool === 'write' && !outcome.isError); + emit({ type: 'done', lastText: session.getLastAssistantText() ?? '', wrote: script.toolName === 'write' && writeSucceeded ? script.path : null }); + } + session.dispose(); + cleanupAgentDir(); + process.exit(0); + } +} catch (error) { + emit({ type: 'fatal', message: error instanceof Error ? error.message : String(error) }); + cleanupAgentDir(); + process.exit(1); +} diff --git a/portable/workspace/agent.test.mjs b/portable/workspace/agent.test.mjs new file mode 100644 index 0000000..1d02c84 --- /dev/null +++ b/portable/workspace/agent.test.mjs @@ -0,0 +1,767 @@ +// Agent adapter checks for the portable workspace prototype. +// +// The fixture transport worker (agent-fixture-worker.mjs) speaks the same +// JSONL protocol as the SDK worker without requiring a built Pi SDK. The +// lifecycle tests repeatedly poll the public events()/HTTP surface to prove +// the success contract on two axes at once — the response events and the +// response state: a worker-reported done is converted to worker_reported_done +// evidence with the payload withheld, and neither a done-typed event nor a +// terminal state is observable before the worker process is actually reaped +// and settled. The canonical done (the retained payload) appears exactly +// once, only after a verified clean reap. The final tests are the gated +// actual-SDK checks: they run the real worker against the built vendored SDK +// and are explicitly skipped (never silently passed) when dist/ is absent. +// See README "Qualification status". +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import http from 'node:http'; +import { execFile } from 'node:child_process'; +import { createWorkspaceServer } from './server.mjs'; +import { createAgentManager } from './agent-manager.mjs'; + +const here = path.dirname(new URL(import.meta.url).pathname); +const fixtureWorker = path.join(here, 'agent-fixture-worker.mjs'); +const sdkDist = path.resolve(here, '../../vendor/pi/packages/coding-agent/dist/index.js'); + +const workspaceDir = async () => fs.mkdtemp(path.join(os.tmpdir(), 'portable-agent-')); + +const managerWith = overrides => + createAgentManager({ workspace: '/tmp', workerPath: fixtureWorker, runTimeoutMs: 60_000, killGraceMs: 1_500, ...overrides }); + +const waitFor = async (manager, id, predicate, label) => { + for (let attempt = 0; attempt < 400; attempt++) { + const record = manager.get(id); + if (record && predicate(record)) return record; + await new Promise(resolve => setTimeout(resolve, 25)); + } + assert.fail(`Timeout waiting for ${label}; state=${manager.get(id)?.state}`); +}; + +// Per-run agent directories the manager creates under the system temp root. +const tempAgentDirs = async () => (await fs.readdir(os.tmpdir())).filter(name => name.startsWith('pi-agent-')).sort(); + +// True when no fixture worker process is left running on this host. +const noFixtureWorkers = () => new Promise(resolve => { + execFile('pgrep', ['-f', fixtureWorker], error => resolve(Boolean(error))); +}); + +const eventTypes = result => result.events.filter(event => event.type === 'agent_event').map(event => event.event); + +// The public terminal states, mirroring the UI's polling contract: a +// consumer that observes one of these stops polling, so none of them may be +// observable before the worker process is actually reaped and settled. +const TERMINAL_RUN_STATES = new Set(['done', 'aborted', 'error', 'timeout', 'killed', 'unavailable']); + +test('agent lifecycle: ordered events, real worker process, bounded reattach', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'write the proof file', script: { proofPath: 'proof.txt', proofContent: 'proof-content\n' } }); + const record = await waitFor(manager, id, r => r.state === 'done' && r.child.exitCode === 0, 'run completion'); + assert.equal(record.child.exitCode, 0, 'fixture worker exits cleanly'); + assert.equal(await fs.readFile(path.join(workspace, 'proof.txt'), 'utf8'), 'proof-content\n'); + const first = manager.events(id, 0); + const types = eventTypes(first); + assert.equal(types[0], 'agent_start'); + assert.ok(types.indexOf('tool_start') < types.indexOf('tool_end'), 'tool events ordered'); + assert.ok(types.indexOf('tool_end') < types.indexOf('agent_settled'), 'settlement after tool work'); + assert.equal(types[types.length - 1], 'agent_settled'); + // Canonical done contract: exactly one done-typed event, appended only + // at the verified clean reap (so it is the final line), carrying the + // worker's retained payload, and preceded by the evidence marker that + // replaced the worker's immediate claim. + const doneEvents = first.events.filter(event => event.type === 'done'); + assert.equal(doneEvents.length, 1, 'exactly one canonical done event'); + assert.equal(first.events[first.events.length - 1].type, 'done', 'the canonical done is appended at the reap, after all worker output'); + assert.equal(doneEvents[0].lastText, 'fixture reply', 'the canonical done carries the retained worker payload'); + const reportedIndex = first.events.findIndex(event => event.type === 'worker_reported_done'); + assert.ok(reportedIndex >= 0 && reportedIndex < first.events.length - 1, 'the worker_reported_done evidence precedes the canonical done'); + assert.deepEqual(Object.keys(first.events[reportedIndex]), ['type'], 'the evidence marker carries no payload; the payload stays private until the reap'); + // Reattach from a mid-stream offset receives only later events. + const firstResult = manager.events(id, 0); + let running = 0; + let offsetAfterToolEnd = 0; + for (const event of firstResult.events) { + running += Buffer.byteLength(JSON.stringify(event)) + 1; + if (event.type === 'agent_event' && event.event === 'tool_end') offsetAfterToolEnd = running; + } + assert.ok(offsetAfterToolEnd > 0); + const tail = manager.events(id, offsetAfterToolEnd); + assert.ok(tail.events.length < firstResult.events.length, 'tail poll is a subset'); + assert.equal(tail.events.some(event => event.type === 'agent_event' && event.event === 'tool_start'), false, 'already-seen events are not repeated'); + assert.equal(manager.list().length, 1); + assert.equal(manager.list()[0].alive, false); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('cancellation: stop aborts a running worker and reaps the process', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'slow run', script: { slow: '1' } }); + await waitFor(manager, id, r => r.state === 'running', 'run start'); + assert.ok(manager.get(id).child.exitCode === null, 'worker alive before stop'); + assert.equal(await manager.stop(id), true); + assert.equal(manager.get(id), null, 'record removed after stop'); + assert.equal((await manager.start({ prompt: 'after stop' })).id.length > 0, true, 'capacity released'); + await waitFor(manager, (await manager.list())[0].id, r => r.state === 'done', 'second run'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('unresponsive worker: timeout escalation marks the run and kills the process', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace, runTimeoutMs: 400, killGraceMs: 400 }); + const child = { pid: null }; + try { + const { id } = await manager.start({ prompt: 'hang', script: { hang: '1' } }); + child.pid = manager.get(id).child.pid; + const record = await waitFor(manager, id, r => r.state === 'timeout' && (r.child.exitCode !== null || r.child.signalCode !== null), 'timeout kill'); + assert.equal(record.state, 'timeout'); + assert.ok(record.child.signalCode !== null || record.child.exitCode !== 0, 'hanging worker is killed, not completed'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('unavailable SDK: probe reports missing build and start is refused', async () => { + const workspace = await workspaceDir(); + const manager = createAgentManager({ workspace, workerPath: path.join(workspace, 'no-such-worker.mjs') }); + try { + const status = await manager.status(); + assert.equal(status.available, false); + assert.equal(status.kind, 'missing'); + assert.match(status.detail, /probe|Pi SDK|spawn/i); + await assert.rejects(manager.start({ prompt: 'x' }), error => error.code === 'AGENT_UNAVAILABLE'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('no ambient configuration: worker env is allowlisted, secrets are not inherited', async () => { + const workspace = await workspaceDir(); + const canary = 'ZAI_TEST_AMBIENT_SECRET'; + process.env[canary] = 'do-not-inherit'; + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'env report', script: { reportEnv: '1' } }); + await waitFor(manager, id, r => r.state === 'done', 'env report'); + const report = manager.events(id, 0).events.find(event => event.type === 'worker_env'); + assert.ok(report, 'worker reported its environment keys'); + const keys = report.env; + for (const key of keys) { + assert.ok(['PATH', 'LANG', 'PI_OFFLINE'].includes(key), `unexpected env key ${key}`); + } + assert.equal(keys.includes(canary), false, 'ambient secret not inherited'); + assert.equal(keys.includes('HOME'), false, 'HOME not forwarded; agentDir is explicit'); + } finally { + delete process.env[canary]; + await manager.close(); + await fs.rm(workspace, {recursive:true,force:true}); + } +}); + +test('event buffer is bounded and offsets report truncation', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace, eventLimit: 4 * 1024 }); + try { + const { id } = await manager.start({ prompt: 'flood', script: { flood: '1' } }); + await waitFor(manager, id, r => r.state === 'done', 'flood completion'); + const result = manager.events(id, 0); + assert.ok(result.truncated, 'early events were dropped'); + assert.ok(result.offset > 0); + const later = manager.events(id, result.offset); + assert.equal(later.events.length, 0); + assert.equal(later.truncated, false); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('HTTP surface: security checks, capacity, validation, and status honesty', async () => { + const workspace = await workspaceDir(); + const agentManager = managerWith({ workspace, maxAgents: 1 }); + const app = await createWorkspaceServer({ workspace, agentManager }); + try { + const status = await (await fetch(app.origin + '/api/status')).json(); + assert.equal(status.agent.kind, 'fixture'); + assert.equal(status.agent.available, true); + const headers = { 'Content-Type': 'application/json', 'X-Workspace-Token': status.csrfToken }; + const request = (url, method = 'GET', data, extra = {}) => + fetch(app.origin + url, { method, headers: { ...headers, ...extra }, ...(data === undefined ? {} : { body: JSON.stringify(data) }) }); + assert.equal((await request('/api/agents', 'POST', { prompt: 'x' }, { Origin: 'https://evil.example' })).status, 403); + assert.equal((await request('/api/agents', 'POST', { prompt: 'x' }, { 'X-Workspace-Token': 'wrong' })).status, 403); + const rebound = await new Promise(resolve => http.get(app.origin + '/api/agents', { headers: { Host: 'evil.example', ...headers } }, res => { res.resume(); resolve(res.statusCode); })); + assert.equal(rebound, 403); + assert.equal((await request('/api/agents', 'POST', {}, { 'Sec-Fetch-Site': 'cross-site' })).status, 403); + assert.equal((await request('/api/agents', 'POST', { prompt: 'x'.repeat(9000) })).status, 400); + assert.equal((await request('/api/agents', 'POST', { prompt: 'ok', script: { surprise: '1' } })).status, 400); + const started = await request('/api/agents', 'POST', { prompt: 'slow', script: { slow: '1' } }); + assert.equal(started.status, 201); + const { id } = await started.json(); + assert.equal((await request('/api/agents', 'POST', { prompt: 'second' })).status, 409); + assert.equal((await request(`/api/agents/${id}/events?after=notanumber`)).status, 400); + let events = null; + for (let attempt = 0; attempt < 100 && !events; attempt++) { + const result = await (await request(`/api/agents/${id}/events?after=0`)).json(); + if (result.events.some(event => event.type === 'agent_event' && event.event === 'agent_start')) events = result; + else await new Promise(resolve => setTimeout(resolve, 25)); + } + assert.ok(events, 'events stream over HTTP'); + const listing = await (await request('/api/agents')).json(); + assert.deepEqual(listing.agents.map(entry => entry.id), [id]); + assert.equal((await request(`/api/agents/${id}`, 'DELETE', {})).status, 200); + assert.equal((await request(`/api/agents/${id}/events`)).status, 404); + } finally { await app.close(); await agentManager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('HTTP surface: unavailable SDK yields an honest 503, not fixture text', async () => { + const workspace = await workspaceDir(); + const app = await createWorkspaceServer({ workspace }); // default worker: real SDK path + try { + const status = await (await fetch(app.origin + '/api/status')).json(); + const exists = await fs.stat(sdkDist).then(() => true, () => false); + assert.equal(status.agent.available, exists, 'status matches the actual built SDK presence'); + const headers = { 'Content-Type': 'application/json', 'X-Workspace-Token': status.csrfToken }; + const response = await fetch(app.origin + '/api/agents', { method: 'POST', headers, body: JSON.stringify({ prompt: 'hello' }) }); + if (!exists) { + assert.equal(response.status, 503); + const data = await response.json(); + assert.match(data.error, /unavailable/i); + assert.ok(data.detail, 'refusal explains why'); + } else { + assert.equal(response.status, 201); + } + } finally { await app.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('probe waits for a complete newline-terminated status line', async () => { + const dir = await workspaceDir(); + const probePath = path.join(dir, 'split-probe.mjs'); + await fs.writeFile(probePath, [ + 'const line = JSON.stringify({ type: "ready", kind: "fixture" });', + 'process.stdout.write(line.slice(0, 24));', + 'setTimeout(() => process.stdout.write(line.slice(24) + "\\n"), 200);', + 'setInterval(() => {}, 60000);', + ].join('\n')); + const manager = managerWith({ workspace: dir, workerPath: probePath }); + try { + const status = await manager.status(); + assert.equal(status.available, true, 'split JSON line is parsed only once the newline arrives'); + assert.equal(status.kind, 'fixture'); + } finally { await manager.close(); await fs.rm(dir, {recursive:true,force:true}); } +}); + +test('probe refuses unbounded non-newline noise instead of buffering it', async () => { + const dir = await workspaceDir(); + const probePath = path.join(dir, 'noise-probe.mjs'); + await fs.writeFile(probePath, [ + 'process.stdout.write("x".repeat(1024 * 1024));', + 'setInterval(() => {}, 60000);', + ].join('\n')); + const manager = managerWith({ workspace: dir, workerPath: probePath, probeTimeoutMs: 30_000 }); + try { + const startedAt = Date.now(); + const status = await manager.status(); + assert.equal(status.available, false); + assert.match(status.detail, /exceeded \d+ bytes/); + assert.ok(Date.now() - startedAt < 10_000, 'refused promptly rather than buffering megabytes'); + } finally { await manager.close(); await fs.rm(dir, {recursive:true,force:true}); } +}); + +test('oversized event frames are refused before retention and the worker is terminated', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace, maxFrameBytes: 8 * 1024, eventLimit: 64 * 1024 }); + try { + const { id } = await manager.start({ prompt: 'oversize', script: { oversize: '1' } }); + const record = await waitFor(manager, id, r => r.state === 'error' && (r.child.exitCode !== null || r.child.signalCode !== null), 'malformed worker exit'); + assert.equal(record.state, 'error', 'a frame-limit violation is terminal'); + const result = manager.events(id, 0); + const note = result.events.find(event => event.type === 'frame_too_large'); + assert.ok(note, 'refusal is visible as a bounded event'); + assert.ok(note.bytes > 8 * 1024 && note.limit === 8 * 1024); + assert.equal(result.events.some(event => event.type === 'agent_event' && event.event === 'text_delta' && (event.delta ?? '').length > 1024), false, 'oversized frame is not retained'); + assert.ok(record.bytes <= 64 * 1024 + 2 * 1024, `retained bytes bounded (got ${record.bytes})`); + for (const event of result.events) { + assert.ok(Buffer.byteLength(JSON.stringify(event)) + 1 <= 8 * 1024 + 512, 'each retained frame is within the limit'); + } + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('late running events cannot revive a terminal run', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'revive attempt', script: { revive: '1' } }); + const record = await waitFor(manager, id, r => r.state === 'done' && r.child.exitCode === 0, 'completion'); + const events = manager.events(id, 0).events; + // The late events land between the evidence marker and the canonical + // done, which is appended at the reap — after everything the worker + // emitted — and remains the only done-typed line in the stream. + const evidenceIndex = events.findIndex(event => event.type === 'worker_reported_done'); + assert.ok(evidenceIndex >= 0, 'the worker done report is retained as evidence'); + assert.ok(events.some((event, index) => event.type === 'state' && event.state === 'running' && index > evidenceIndex), 'fixture emitted a late running event after its done report'); + const doneEvents = events.filter(event => event.type === 'done'); + assert.equal(doneEvents.length, 1, 'exactly one canonical done'); + assert.equal(events[events.length - 1].type, 'done', 'the canonical done is appended at the reap, after the late events'); + assert.equal(doneEvents[0].lastText, 'fixture reply', 'the canonical done carries the retained payload'); + assert.equal(record.state, 'done', 'final state stays monotonic'); + assert.equal(manager.list()[0].alive, false); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('concurrent starts: capacity is reserved synchronously; at most maxAgents accepted', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace, maxAgents: 2 }); + const dirsBefore = await tempAgentDirs(); + try { + // Five parallel starts against a capacity of two: the reservation + // happens before any await (including agentDir creation), so exactly + // two are accepted no matter how the interleaving lands. + const attempts = await Promise.allSettled(Array.from({ length: 5 }, () => + manager.start({ prompt: 'race', script: { slow: '1' } }))); + const accepted = attempts.filter(result => result.status === 'fulfilled'); + const rejected = attempts.filter(result => result.status === 'rejected'); + assert.equal(accepted.length, 2, 'exactly the capacity is accepted'); + assert.equal(rejected.length, 3); + for (const result of rejected) assert.equal(result.reason.code, 'AGENT_CAPACITY'); + // No child:null records are ever exposed, and both live children hold capacity. + const listing = manager.list(); + assert.equal(listing.length, 2); + for (const entry of listing) { + assert.equal(entry.alive, true); + assert.ok(manager.get(entry.id).child, 'published records always carry their child'); + } + await assert.rejects(manager.start({ prompt: 'third' }), error => error.code === 'AGENT_CAPACITY'); + // Capacity recovers only once the children are actually reaped. + for (const result of accepted) await waitFor(manager, result.value.id, r => r.state === 'done' && r.reaped, 'race run completion'); + const { id } = await manager.start({ prompt: 'recovered' }); + await waitFor(manager, id, r => r.state === 'done' && r.reaped, 'capacity recovery'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } + assert.deepEqual(await tempAgentDirs(), dirsBefore, 'no stray per-run temp directories'); +}); + +test('reported done is not terminal success: repeated polls during the gap never observe a done event or terminal state (doneThenHang)', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace, maxAgents: 1, runTimeoutMs: 1_000, killGraceMs: 300 }); + try { + const { id } = await manager.start({ prompt: 'done then hang', script: { doneThenHang: '1' } }); + // Poll the public events() surface repeatedly across the gap between the + // worker's done report and its reap: the report is visible only as a + // worker_reported_done evidence marker (payload withheld), and BOTH the + // event stream and the authoritative state must stay free of any + // done-typed/terminal claim while the worker process is still live. + let sawReported = false; + let gapPolls = 0; + let offset = 0; + for (let attempt = 0; attempt < 600; attempt++) { + const record = manager.get(id); + if (!record) break; + const reapedNow = record.reaped; + const result = manager.events(id, offset); + offset = result.offset; + if (!reapedNow) { + if (result.events.some(event => event.type === 'worker_reported_done')) sawReported = true; + if (sawReported) { + gapPolls += 1; + // The only state change that may appear before the reap is the + // manager-initiated 'timeout' (the kill it initiates): the + // worker's own done report must never produce any terminal + // state, least of all 'done'. + assert.ok(result.state === 'running' || result.state === 'timeout', `authoritative state stays nonterminal on the worker's own claim (got ${result.state})`); + assert.notEqual(result.state, 'done', 'the done report never yields the terminal success state before reap'); + assert.equal(result.events.some(event => event.type === 'done'), false, 'no canonical done event is observable before the reap'); + } + if (sawReported && gapPolls === 1) { + // First observation of the gap: the previous lifecycle protections + // still hold while the done-reporting worker remains live. + assert.ok(!record.reaped && record.child.exitCode === null, 'the worker process is still live after reporting done'); + assert.equal(manager.list()[0].alive, true, 'listing reports process truth, not the protocol claim'); + await assert.rejects(manager.start({ prompt: 'second' }), error => error.code === 'AGENT_CAPACITY', 'a live done-reporting child still occupies capacity'); + } + } + if (reapedNow) break; + await new Promise(resolve => setTimeout(resolve, 15)); + } + assert.ok(sawReported, 'the worker reported done; the claim is visible as evidence'); + assert.ok(gapPolls >= 5, `the provisional-done gap was observed by repeated polls, not only the final outcome (got ${gapPolls})`); + // The run timeout still governs the process: the hung worker is killed. + const finished = await waitFor(manager, id, r => r.state === 'timeout' && r.reaped, 'timeout kill of a done-reporting worker'); + assert.ok(finished.child.signalCode !== null || finished.child.exitCode !== 0, 'the worker was actually killed, not completed'); + // Timeout precedence over the done claim, on the event axis too: the + // final stream has the evidence marker but no canonical done ever. + const final = manager.events(id, 0); + assert.equal(final.state, 'timeout'); + assert.equal(final.events.filter(event => event.type === 'done').length, 0, 'timeout precedence: no canonical done is ever appended'); + assert.equal(final.events.filter(event => event.type === 'worker_reported_done').length, 1, 'the provisional claim is retained as exactly one evidence marker'); + // After the reap the slot is free again. + const { id: next } = await manager.start({ prompt: 'recovered' }); + await waitFor(manager, next, r => r.state === 'done' && r.reaped, 'capacity recovery'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('delayed nonzero exit: done stays provisional until the reap settles it as error', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'done then delayed nonzero exit', script: { doneThenExit: '7', exitDelayMs: '800' } }); + // Repeatedly poll across the 800 ms gap between the done report and the + // nonzero exit: every poll must show no done-typed event and a + // nonterminal state; success is settled only after the exit is reaped. + let sawReported = false; + let gapPolls = 0; + let offset = 0; + for (let attempt = 0; attempt < 400; attempt++) { + const record = manager.get(id); + if (!record) break; + const reapedNow = record.reaped; + const result = manager.events(id, offset); + offset = result.offset; + if (!reapedNow) { + if (result.events.some(event => event.type === 'worker_reported_done')) sawReported = true; + if (sawReported) { + gapPolls += 1; + assert.equal(result.state, 'running', 'authoritative state stays nonterminal between the done report and the nonzero exit'); + assert.equal(result.events.some(event => event.type === 'done'), false, 'no canonical done event is observable before the nonzero exit is reaped'); + } + } + if (reapedNow) break; + await new Promise(resolve => setTimeout(resolve, 15)); + } + assert.ok(sawReported, 'the worker reported done'); + assert.ok(gapPolls >= 5, `the gap was observed by repeated polls, not only the final outcome (got ${gapPolls})`); + const settled = await waitFor(manager, id, r => r.state === 'error' && r.reaped, 'delayed nonzero settlement'); + assert.equal(settled.child.exitCode, 7); + const final = manager.events(id, 0); + assert.equal(final.state, 'error', 'the response state is authoritative'); + assert.equal(final.events.filter(event => event.type === 'done').length, 0, 'no canonical done exists for a nonzero exit'); + assert.ok(final.events.some(event => event.type === 'worker_reported_done'), 'the protocol done is retained as provisional evidence'); + const exit = final.events.find(event => event.type === 'worker_exit'); + assert.ok(exit, 'the unclean exit is recorded'); + assert.equal(exit.code, 7); + assert.match(exit.detail, /uncleanly/); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('delayed clean exit: the canonical done appears exactly once, only at the verified reap', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'done then delayed clean exit', script: { doneThenExit0: '1', exitDelayMs: '800' } }); + // Repeated polls across the delayed-exit window must observe the + // worker_reported_done evidence but never a done-typed event and never + // a terminal state; the canonical done is appended exactly once, at the + // clean reap, with the retained payload. + let sawReported = false; + let gapPolls = 0; + let offset = 0; + for (let attempt = 0; attempt < 400; attempt++) { + const record = manager.get(id); + if (!record) break; + const reapedNow = record.reaped; + const result = manager.events(id, offset); + offset = result.offset; + if (!reapedNow) { + if (result.events.some(event => event.type === 'worker_reported_done')) sawReported = true; + if (sawReported) { + gapPolls += 1; + assert.equal(result.state, 'running', 'state stays nonterminal across the done-report-to-exit gap'); + assert.equal(result.events.some(event => event.type === 'done'), false, 'no canonical done event is observable before the clean exit is reaped'); + } + } + if (reapedNow) break; + await new Promise(resolve => setTimeout(resolve, 15)); + } + assert.ok(sawReported, 'the worker reported done; the evidence marker was observed'); + assert.ok(gapPolls >= 5, `the delayed-exit window was observed by repeated polls (got ${gapPolls})`); + await waitFor(manager, id, r => r.state === 'done' && r.reaped, 'clean settlement'); + const final = manager.events(id, 0); + assert.equal(final.state, 'done'); + const doneEvents = final.events.filter(event => event.type === 'done'); + assert.equal(doneEvents.length, 1, 'exactly one canonical done after the clean reap'); + assert.equal(final.events[final.events.length - 1].type, 'done', 'the canonical done is the final event, appended at the reap'); + assert.equal(doneEvents[0].lastText, 'fixture reply', 'the canonical done carries the privately retained payload'); + assert.equal(manager.list()[0].alive, false, 'the worker is reaped'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('worker-reported aborted after done outranks the completion claim: no canonical done', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'done then aborted', script: { doneThenAbort: '1' } }); + const record = await waitFor(manager, id, r => r.state === 'aborted' && r.reaped, 'aborted settlement'); + assert.equal(record.child.exitCode, 0, 'the worker exited cleanly, but aborted outranks the done claim'); + const final = manager.events(id, 0); + assert.equal(final.state, 'aborted'); + assert.equal(final.events.filter(event => event.type === 'done').length, 0, 'no canonical done is appended when the worker retracted its completion'); + assert.equal(final.events.filter(event => event.type === 'worker_reported_done').length, 1, 'the provisional claim is retained as evidence'); + assert.equal(final.events[final.events.length - 1].type, 'aborted', 'the worker-reported aborted outcome is the terminal event'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('HTTP events surface: a provisional done never reads as terminal success before reap', async () => { + const workspace = await workspaceDir(); + const agentManager = managerWith({ workspace }); + const app = await createWorkspaceServer({ workspace, agentManager }); + try { + const status = await (await fetch(app.origin + '/api/status')).json(); + const headers = { 'Content-Type': 'application/json', 'X-Workspace-Token': status.csrfToken }; + const request = (url, method = 'GET', data) => + fetch(app.origin + url, { method, headers, ...(data === undefined ? {} : { body: JSON.stringify(data) }) }); + const started = await request('/api/agents', 'POST', { prompt: 'delayed nonzero over HTTP', script: { doneThenExit: '3', exitDelayMs: '800' } }); + assert.equal(started.status, 201); + const { id } = await started.json(); + // Poll the exact wire surface the UI polls: while the worker_reported_done + // evidence marker is already in the stream, neither a done-typed event + // nor a terminal state may be observable until the reap settles the run + // — for this worker, as an error. + let sawReported = false; + let gapPolls = 0; + let offset = 0; + let finalState = null; + for (let attempt = 0; attempt < 200; attempt++) { + const result = await (await request(`/api/agents/${id}/events?after=${offset}`)).json(); + offset = result.offset; + if (result.events.some(event => event.type === 'worker_reported_done')) sawReported = true; + if (TERMINAL_RUN_STATES.has(result.state)) { finalState = result.state; break; } + if (sawReported) { + gapPolls += 1; + assert.equal(result.state, 'running', 'no terminal state is observable over HTTP before the worker process is reaped'); + assert.equal(result.events.some(event => event.type === 'done'), false, 'no canonical done event is observable over HTTP before the reap'); + } + await new Promise(resolve => setTimeout(resolve, 15)); + } + assert.equal(finalState, 'error', 'the run settles as error, the only legitimate terminal outcome for this worker'); + assert.ok(sawReported, 'the provisional done report was visible as evidence'); + assert.ok(gapPolls >= 5, `the provisional-done gap was observed over HTTP by repeated polls (got ${gapPolls})`); + const all = await (await request(`/api/agents/${id}/events?after=0`)).json(); + assert.equal(all.state, 'error', 'response.state is authoritative'); + assert.equal(all.events.filter(event => event.type === 'done').length, 0, 'no canonical done event exists in the final stream'); + assert.ok(all.events.some(event => event.type === 'worker_reported_done'), 'the worker_reported_done evidence marker remains in the stream'); + const exit = all.events.find(event => event.type === 'worker_exit'); + assert.ok(exit, 'the unclean exit is recorded'); + assert.equal(exit.code, 3); + assert.match(exit.detail, /uncleanly/); + const listing = await (await request('/api/agents')).json(); + assert.equal(listing.agents[0].alive, false, 'the worker is reaped, not merely claimed done'); + } finally { await app.close(); await agentManager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('done followed by a nonzero exit is an error, not success', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'done then bad exit', script: { doneThenExit: '7' } }); + const record = await waitFor(manager, id, r => r.state === 'error' && r.reaped, 'done-then-nonzero settlement'); + assert.equal(record.child.exitCode, 7); + const events = manager.events(id, 0).events; + assert.equal(events.filter(event => event.type === 'done').length, 0, 'no canonical done is ever appended for an unclean exit'); + assert.ok(events.some(event => event.type === 'worker_reported_done'), 'the protocol done is retained as evidence'); + const exit = events.find(event => event.type === 'worker_exit'); + assert.ok(exit, 'the unclean exit is recorded'); + assert.equal(exit.code, 7); + assert.match(exit.detail, /uncleanly/); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('exit 0 without reporting completion is an error, not done', async () => { + const workspace = await workspaceDir(); + const manager = managerWith({ workspace }); + try { + const { id } = await manager.start({ prompt: 'silent exit', script: { exit0NoDone: '1' } }); + await waitFor(manager, id, r => r.state === 'error' && r.reaped, 'silent-exit settlement'); + const events = manager.events(id, 0).events; + assert.equal(events.some(event => event.type === 'done'), false, 'no completion was ever reported'); + assert.equal(events.some(event => event.type === 'worker_reported_done'), false, 'no done report evidence exists either'); + const exit = events.find(event => event.type === 'worker_exit'); + assert.ok(exit, 'the exit is recorded'); + assert.equal(exit.code, 0); + assert.match(exit.detail, /without reporting completion/); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('racing start and close: in-flight starts settle, no stray children or temp dirs', async () => { + const dir = await workspaceDir(); + const dirsBefore = await tempAgentDirs(); + // A: deterministic race — a slow probe keeps the start in flight while + // close() begins; the start must roll back and reject, not spawn. + const probePath = path.join(dir, 'slow-probe.mjs'); + await fs.writeFile(probePath, [ + 'setTimeout(() => { process.stdout.write(JSON.stringify({ type: "ready", kind: "fixture" }) + "\\n"); }, 400);', + 'setInterval(() => {}, 60000);', + ].join('\n')); + const racing = managerWith({ workspace: dir, workerPath: probePath, probeTimeoutMs: 30_000 }); + const attempt = racing.start({ prompt: 'race close' }); + await new Promise(resolve => setTimeout(resolve, 100)); // inside the probe await + await racing.close(); + await assert.rejects(attempt, error => error.code === 'AGENT_CLOSING', 'in-flight start is rejected once closing begins'); + await assert.rejects(racing.start({ prompt: 'after close' }), error => error.code === 'AGENT_CLOSING', 'starts after close are rejected'); + assert.deepEqual(racing.list(), []); + // B: close() stops and reaps a running agent and awaits its cleanup. + const manager = managerWith({ workspace: dir }); + try { + const { id } = await manager.start({ prompt: 'slow', script: { slow: '1' } }); + await waitFor(manager, id, r => r.state === 'running', 'run start'); + await manager.close(); + assert.deepEqual(manager.list(), [], 'close removes every record'); + assert.equal(await noFixtureWorkers(), true, 'no stray worker processes survive close'); + } finally { await manager.close(); } + await fs.rm(dir, {recursive:true,force:true}); + assert.deepEqual(await tempAgentDirs(), dirsBefore, 'no stray per-run temp directories'); +}); + +test('terminal reattach: listing exposes surviving sessions after a page reload', async () => { + const workspace = await workspaceDir(); + const app = await createWorkspaceServer({ workspace }); + try { + const status = await (await fetch(app.origin + '/api/status')).json(); + const headers = { 'Content-Type': 'application/json', 'X-Workspace-Token': status.csrfToken }; + const first = await (await fetch(app.origin + '/api/terminals', { method: 'POST', headers, body: '{}' })).json(); + const second = await (await fetch(app.origin + '/api/terminals', { method: 'POST', headers, body: '{}' })).json(); + const listing = await (await fetch(app.origin + '/api/terminals')).json(); + assert.deepEqual(listing.sessions.map(session => session.id).sort(), [first.id, second.id].sort()); + await fetch(app.origin + `/api/terminals/${first.id}`, { method: 'DELETE', headers, body: '{}' }); + const after = await (await fetch(app.origin + '/api/terminals')).json(); + assert.deepEqual(after.sessions.map(session => session.id), [second.id]); + assert.equal((await fetch(app.origin + `/api/terminals/${second.id}/output?after=0`)).status, 200); + } finally { await app.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +// Gated actual-SDK smoke check. This test runs agent-worker.mjs (the real +// vendored Pi SDK, scripted local model) through the manager. When the SDK is +// not built it is explicitly SKIPPED — it never reports an actual-SDK pass on +// the strength of the fixture transport. +test('actual SDK: workspace auth/models canaries ignored, per-run temp agentDir, real tool file and event order', async t => { + const built = await fs.stat(sdkDist).then(() => true, () => false); + if (!built) return t.skip(`Pi SDK dist not built at ${sdkDist}; actual-SDK smoke check skipped (not passed)`); + + const workspace = await workspaceDir(); + const canaryKey = 'ZAI-CANARY-KEY-DO-NOT-USE'; + const canaryAuth = JSON.stringify({ 'canary-provider': { type: 'api_key', key: canaryKey } }, null, 2); + const canaryModels = JSON.stringify({ + providers: { 'canary-provider': { name: 'Canary', baseUrl: 'https://canary.example.invalid', api: 'openai-compliant', models: [{ id: 'canary-model', name: 'Canary model' }] } }, + }, null, 2); + await fs.mkdir(path.join(workspace, '.pi-agent'), { recursive: true }); + // A symlinked auth.json plus a plain models.json with canary values, both + // inside the workspace: if the SDK consulted them the run would pick up the + // canary provider/model and credential. + await fs.writeFile(path.join(workspace, 'canary-auth-target.json'), canaryAuth); + await fs.symlink('canary-auth-target.json', path.join(workspace, '.pi-agent', 'auth.json')); + await fs.writeFile(path.join(workspace, '.pi-agent', 'models.json'), canaryModels); + + const manager = createAgentManager({ workspace, runTimeoutMs: 120_000, killGraceMs: 2_000 }); + try { + const { id } = await manager.start({ prompt: 'write the proof file', script: { path: 'sdk-proof.txt', content: 'actual-sdk\nproof\n' } }); + const record = await waitFor(manager, id, r => r.state === 'done' && r.child.exitCode === 0, 'actual SDK run completion'); + assert.equal(await fs.readFile(path.join(workspace, 'sdk-proof.txt'), 'utf8'), 'actual-sdk\nproof\n', 'the real SDK write tool produced the file'); + + const result = manager.events(id, 0); + const types = eventTypes(result); + assert.equal(types[0], 'agent_start'); + assert.ok(types.indexOf('tool_start') < types.indexOf('tool_end'), 'tool events ordered'); + assert.ok(types.indexOf('tool_end') < types.indexOf('agent_settled'), 'settlement after tool work'); + const toolStart = result.events.find(event => event.type === 'agent_event' && event.event === 'tool_start'); + assert.equal(toolStart.tool, 'write'); + // Canonical done contract against the actual SDK worker: exactly one + // done-typed line, appended at the verified reap (it is the final + // event), preceded by the payload-less evidence marker. + const doneEvents = result.events.filter(event => event.type === 'done'); + assert.equal(doneEvents.length, 1, 'exactly one canonical done event'); + assert.equal(result.events[result.events.length - 1].type, 'done', 'the canonical done is appended at the reap'); + assert.equal(doneEvents[0].wrote, 'sdk-proof.txt', 'the canonical done carries the worker payload'); + assert.ok(result.events.some(event => event.type === 'worker_reported_done'), 'the provisional report is retained as evidence'); + + // Isolation evidence from the SDK runtime itself (not mocked call options). + const diag = result.events.find(event => event.type === 'sdk_diag'); + assert.ok(diag, 'worker reports post-run SDK diagnostics'); + assert.deepEqual(diag.credentialProviders, ['abalone-scripted'], 'in-memory AuthStorage is the live credential store'); + assert.ok(diag.models.every(entry => !entry.includes('canary')), 'canary provider/model from a read models.json is absent'); + assert.equal(JSON.stringify(result.events).includes('canary'), false, 'no canary material anywhere in the event stream'); + assert.equal(JSON.stringify(result.events).includes(canaryKey), false, 'canary key never surfaces'); + + // The poisoned files were never read or rewritten. + assert.equal(await fs.readFile(path.join(workspace, '.pi-agent', 'models.json'), 'utf8'), canaryModels, 'models.json untouched'); + assert.equal(await fs.readFile(path.join(workspace, 'canary-auth-target.json'), 'utf8'), canaryAuth, 'symlinked auth target untouched'); + + // Per-run agentDir: fresh 0700 temp dir outside the workspace, cleaned after exit. + const agentDir = record.agentDir; + assert.ok(agentDir.startsWith(path.join(os.tmpdir(), 'pi-agent-')), 'agentDir is a fresh temp directory'); + assert.ok(!agentDir.startsWith(workspace), 'agentDir is outside the workspace'); + let removed = false; + for (let attempt = 0; attempt < 160 && !removed; attempt++) { + removed = !(await fs.stat(agentDir).then(() => true, () => false)); + if (!removed) await new Promise(resolve => setTimeout(resolve, 25)); + } + assert.ok(removed, 'the created agentDir is removed after the worker exits'); + const entries = (await fs.readdir(workspace)).sort(); + assert.deepEqual(entries, ['.pi-agent', 'canary-auth-target.json', 'sdk-proof.txt'], 'no stray agent state written into the workspace'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +// Gated actual-SDK negatives: the scripted proof transport must never report +// success when the real SDK tool execution failed, was an unknown tool, or +// was only a read. See README "Scripted-proof tool outcome policy". +test('actual SDK: failed tool executions produce errors, never claimed writes', async t => { + const built = await fs.stat(sdkDist).then(() => true, () => false); + if (!built) return t.skip(`Pi SDK dist not built at ${sdkDist}; actual-SDK negative checks skipped (not passed)`); + + const workspace = await workspaceDir(); + const manager = createAgentManager({ workspace, runTimeoutMs: 120_000, killGraceMs: 2_000 }); + try { + // Negative: the real SDK write tool cannot write to a directory path. + const { id: writeId } = await manager.start({ prompt: 'failed write', script: { toolName: 'write', path: '.', content: 'no\n' } }); + const writeRecord = await waitFor(manager, writeId, r => r.state === 'error' && r.reaped, 'write-directory refusal'); + assert.equal(writeRecord.child.exitCode, 1, 'worker exits nonzero on a failed planned tool'); + const writeEvents = manager.events(writeId, 0).events; + const writeToolEnd = writeEvents.find(event => event.type === 'agent_event' && event.event === 'tool_end'); + assert.equal(writeToolEnd.isError, true, 'the real SDK write tool failed'); + const writeFailed = writeEvents.find(event => event.type === 'tool_failed'); + assert.ok(writeFailed, 'the captured tool failure is reported'); + assert.equal(writeFailed.tool, 'write'); + assert.equal(writeEvents.some(event => event.type === 'done'), false, 'no success is claimed'); + + // Negative: reading a file that does not exist fails the run. + const { id: readId } = await manager.start({ prompt: 'missing read', script: { toolName: 'read', path: 'no-such-file.txt' } }); + await waitFor(manager, readId, r => r.state === 'error' && r.reaped, 'missing-read refusal'); + const readEvents = manager.events(readId, 0).events; + const readToolEnd = readEvents.find(event => event.type === 'agent_event' && event.event === 'tool_end'); + assert.equal(readToolEnd.tool, 'read'); + assert.equal(readToolEnd.isError, true, 'the real SDK read tool failed'); + assert.ok(readEvents.some(event => event.type === 'tool_failed')); + assert.equal(readEvents.some(event => event.type === 'done'), false, 'no success is claimed'); + + // Negative: an unknown planned tool is refused outright. + const { id: badId } = await manager.start({ prompt: 'invalid tool', script: { toolName: 'bogus' } }); + await waitFor(manager, badId, r => r.state === 'error' && r.reaped, 'invalid-tool refusal'); + const badEvents = manager.events(badId, 0).events; + const fatal = badEvents.find(event => event.type === 'fatal'); + assert.ok(fatal, 'the invalid planned tool is rejected with an explicit error'); + assert.match(fatal.message, /read.*write|write.*read/); + assert.equal(badEvents.some(event => event.type === 'done'), false, 'no success is claimed'); + + assert.deepEqual((await fs.readdir(workspace)).sort(), [], 'no tool file was created by any negative run'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); + +test('actual SDK: a successful read reports no wrote claim', async t => { + const built = await fs.stat(sdkDist).then(() => true, () => false); + if (!built) return t.skip(`Pi SDK dist not built at ${sdkDist}; actual-SDK read check skipped (not passed)`); + + const workspace = await workspaceDir(); + await fs.writeFile(path.join(workspace, 'note.txt'), 'actual-sdk note\n'); + const manager = createAgentManager({ workspace, runTimeoutMs: 120_000, killGraceMs: 2_000 }); + try { + const { id } = await manager.start({ prompt: 'read the note', script: { toolName: 'read', path: 'note.txt', finalText: 'read it' } }); + await waitFor(manager, id, r => r.state === 'done' && r.reaped && r.child.exitCode === 0, 'read-only run completion'); + const events = manager.events(id, 0).events; + const toolEnd = events.find(event => event.type === 'agent_event' && event.event === 'tool_end'); + assert.equal(toolEnd.tool, 'read'); + assert.equal(toolEnd.isError, false); + const done = events.find(event => event.type === 'done'); + assert.ok(done, 'a successful read still completes the run'); + assert.equal(done.wrote, null, 'a read never claims a wrote path'); + assert.equal(events.filter(event => event.type === 'done').length, 1, 'exactly one canonical done, appended at the reap'); + const evidenceIndex = events.findIndex(event => event.type === 'worker_reported_done'); + assert.ok(evidenceIndex >= 0, 'the provisional report is retained as evidence'); + assert.ok(events.indexOf(done) > evidenceIndex, 'the evidence marker precedes the canonical done'); + assert.equal(await fs.readFile(path.join(workspace, 'note.txt'), 'utf8'), 'actual-sdk note\n', 'the file is untouched'); + assert.deepEqual((await fs.readdir(workspace)).sort(), ['note.txt'], 'nothing was written'); + } finally { await manager.close(); await fs.rm(workspace, {recursive:true,force:true}); } +}); diff --git a/portable/workspace/server.mjs b/portable/workspace/server.mjs new file mode 100644 index 0000000..431e7e2 --- /dev/null +++ b/portable/workspace/server.mjs @@ -0,0 +1,133 @@ +import http from 'node:http'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import { spawn } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; +import { createAgentManager } from './agent-manager.mjs'; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const limit = 256 * 1024; +const agentScriptKeys = ['toolName', 'path', 'content', 'finalText', 'reportEnv', 'slow', 'flood', 'hang', 'proofPath', 'proofContent', 'oversize', 'revive', 'doneThenHang', 'doneThenExit', 'doneThenExit0', 'doneThenAbort', 'exitDelayMs', 'exit0NoDone']; +export async function createWorkspaceServer({ workspace, port = 0, agentManager }) { + if (!workspace || !path.isAbsolute(workspace)) throw new Error('An explicit absolute disposable workspace is required'); + const root = await fs.realpath(workspace); + if (!(await fs.stat(root)).isDirectory() || root === '/') throw new Error('Workspace must be a directory other than /'); + const token = crypto.randomBytes(32).toString('hex'); + const terminals = new Map(); + const agents = agentManager ?? createAgentManager({ workspace: root }); + let origin; + const send = (res, code, data) => { res.writeHead(code, {'content-type':'application/json'}); res.end(JSON.stringify(data)); }; + const body = async req => { + let value = ''; + for await (const chunk of req) { value += chunk; if (Buffer.byteLength(value) > 16384) throw new Error('Request too large'); } + return JSON.parse(value || '{}'); + }; + const stop = terminal => { terminal.child.stdin.end(); }; + const server = http.createServer(async (req, res) => { + res.setHeader('Cache-Control', 'no-store'); + res.setHeader('X-Content-Type-Options', 'nosniff'); + res.setHeader('Content-Security-Policy', "default-src 'self'; script-src 'self'; style-src 'self'; connect-src 'self'; img-src 'self' data:; frame-ancestors 'none'; base-uri 'none'; form-action 'self'"); + try { + // Host checks prevent DNS rebinding; Origin/Fetch Metadata and token reject cross-site commands. + if (req.headers.host !== new URL(origin).host || (req.headers.origin && req.headers.origin !== origin) || req.headers['sec-fetch-site'] === 'cross-site') return send(res, 403, {error:'Origin refused'}); + const url = new URL(req.url, origin); + if (!['GET', 'HEAD'].includes(req.method)) { + if (req.headers['x-workspace-token'] !== token || req.headers['content-type'] !== 'application/json') return send(res, 403, {error:'Workspace token and JSON required'}); + } + if (req.method === 'GET' && url.pathname === '/api/status') { + const agent = await agents.status().catch(error => ({ available: false, kind: 'missing', detail: error.message })); + return send(res, 200, { terminalAvailable:true, csrfToken:token, workspace:root, qualification:'local-development', agent }); + } + if (req.method === 'GET' && url.pathname === '/api/terminals') { + return send(res, 200, { sessions: [...terminals.entries()].map(([id, terminal]) => ({ id, exited: terminal.exited })) }); + } + if (req.method === 'POST' && url.pathname === '/api/agents') { + const { prompt, script } = await body(req); + if (typeof prompt !== 'string' || !prompt.trim() || Buffer.byteLength(prompt) > 8192) return send(res, 400, {error:'Prompt must be a non-empty string of at most 8192 bytes'}); + if (script !== undefined && (typeof script !== 'object' || script === null || Array.isArray(script))) return send(res, 400, {error:'Invalid script'}); + if (script) { + const keys = Object.keys(script); + if (keys.length > 6 || keys.some(key => !agentScriptKeys.includes(key) || typeof script[key] !== 'string' || Buffer.byteLength(script[key]) > 2000)) { + return send(res, 400, {error:'Invalid script fields'}); + } + } + try { + const { id } = await agents.start({ prompt, script }); + return send(res, 201, { id }); + } catch (error) { + if (error.code === 'AGENT_CAPACITY') return send(res, 409, { error: error.message }); + if (error.code === 'AGENT_CLOSING') return send(res, 503, { error: 'Agent manager is closing', detail: error.message }); + return send(res, 503, { error: 'Agent SDK unavailable', detail: error.message }); + } + } + if (req.method === 'GET' && url.pathname === '/api/agents') return send(res, 200, { agents: agents.list() }); + const agentMatch = /^\/api\/agents\/([a-f0-9-]+)(\/events)?$/.exec(url.pathname); + if (agentMatch) { + if (req.method === 'GET' && agentMatch[2] === '/events') { + const result = agents.events(agentMatch[1], Number(url.searchParams.get('after') || 0)); + if (!result) return send(res, 404, {error:'Agent run not found'}); + return send(res, 200, result); + } + if (req.method === 'DELETE' && agentMatch[2] === undefined) { + const removed = await agents.stop(agentMatch[1]); + return send(res, removed ? 200 : 404, removed ? {ok:true} : {error:'Agent run not found'}); + } + } + if (req.method === 'POST' && url.pathname === '/api/terminals') { + await body(req); + if (terminals.size >= 8) return send(res, 409, {error:'Close a terminal before opening another'}); + const id = crypto.randomUUID(); + const child = spawn('python3', ['-I', path.join(here, 'terminal_bridge.py'), root], {stdio:['pipe','pipe','pipe'], env:{PATH:process.env.PATH}}); + const terminal = {child, output:'', base:0, exited:false}; + child.stdout.setEncoding('utf8'); + const append = data => { terminal.output += data; if (terminal.output.length > limit) { const remove = terminal.output.length - limit; terminal.output = terminal.output.slice(remove); terminal.base += remove; } }; + child.stdout.on('data', append); + child.stderr.on('data', data => append(String(data))); + child.stdin.on('error', () => {}); + child.on('error', error => { append(`Terminal unavailable: ${error.message}`); terminal.exited = true; }); + child.on('close', () => { terminal.exited = true; }); + terminals.set(id, terminal); + return send(res, 201, {id}); + } + const match = /^\/api\/terminals\/([a-f0-9-]+)(?:\/(input|output))?$/.exec(url.pathname); + if (match) { + const terminal = terminals.get(match[1]); + if (!terminal) return send(res, 404, {error:'Terminal not found'}); + if (req.method === 'GET' && match[2] === 'output') { + const after = Number(url.searchParams.get('after') || 0); + if (!Number.isSafeInteger(after) || after < 0) return send(res, 400, {error:'Invalid offset'}); + return send(res, 200, {output:terminal.output.slice(Math.max(0, after - terminal.base)), offset:terminal.base + terminal.output.length, exited:terminal.exited, truncated:after < terminal.base}); + } + if (req.method === 'POST' && match[2] === 'input') { + const {input} = await body(req); + if (typeof input !== 'string' || Buffer.byteLength(input) > 8192) return send(res, 400, {error:'Invalid input'}); + if (terminal.exited || terminal.child.stdin.writableLength > 65536) return send(res, 409, {error:'Terminal is closed or busy'}); + terminal.child.stdin.write(JSON.stringify({input}) + '\n'); + return send(res, 200, {ok:true}); + } + if (req.method === 'DELETE' && !match[2]) { stop(terminal); terminals.delete(match[1]); return send(res, 200, {ok:true}); } + } + if (req.method !== 'GET' || url.pathname.startsWith('/api/')) return send(res, 404, {error:'Not found'}); + const names = {'/':'index.html','/index.html':'index.html','/app.js':'app.js','/style.css':'style.css','/styles.css':'styles.css'}; + const name = names[url.pathname]; + if (!name) return send(res, 404, {error:'Not found'}); + const data = await fs.readFile(path.join(here, 'ui', name)); + res.setHeader('Content-Type', name.endsWith('.html') ? 'text/html; charset=utf-8' : name.endsWith('.js') ? 'text/javascript; charset=utf-8' : 'text/css; charset=utf-8'); + res.end(data); + } catch (error) { send(res, error.code === 'ENOENT' ? 404 : 400, {error:error.code === 'ENOENT' ? 'Not found' : 'Request refused'}); } + }); + await new Promise(resolve => server.listen(port, '127.0.0.1', resolve)); + origin = `http://127.0.0.1:${server.address().port}`; + return {origin, async close() { for (const terminal of terminals.values()) stop(terminal); await agents.close(); await new Promise(resolve => server.close(resolve)); }}; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + const args = process.argv.slice(2); + const workspace = args[args.indexOf('--workspace') + 1]; + if (!args.includes('--workspace')) throw new Error('Usage: node server.mjs --workspace /absolute/disposable/folder [--port 8765]'); + const port = args.includes('--port') ? Number(args[args.indexOf('--port') + 1]) : 8765; + const app = await createWorkspaceServer({workspace, port}); + console.log(`Local prototype: ${app.origin}\nReal terminals run as your account. The agent adapter uses a local scripted model (no provider inference) when the Pi SDK is built; otherwise it reports unavailable. Funding stays a fixture. No sandbox or Tor enforcement.`); + for (const signal of ['SIGINT','SIGTERM']) process.once(signal, async () => { await app.close(); process.exit(0); }); +} diff --git a/portable/workspace/server.test.mjs b/portable/workspace/server.test.mjs new file mode 100644 index 0000000..521e533 --- /dev/null +++ b/portable/workspace/server.test.mjs @@ -0,0 +1,48 @@ +import {test} from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import http from 'node:http'; +import {createWorkspaceServer} from './server.mjs'; + +test('PTY persists shell state, accepts interrupt, and refuses foreign commands', async () => { + const workspace = await fs.mkdtemp(path.join(os.tmpdir(), 'portable-workspace-')); + const app = await createWorkspaceServer({workspace}); + try { + const status = await (await fetch(app.origin + '/api/status')).json(); + const headers = {'Content-Type':'application/json','X-Workspace-Token':status.csrfToken}; + const request = (url, method = 'GET', data, extra = {}) => fetch(app.origin + url, {method, headers:{...headers,...extra}, ...(data === undefined ? {} : {body:JSON.stringify(data)})}); + assert.equal((await request('/api/terminals','POST',{}, {Origin:'https://evil.example'})).status,403); + assert.equal((await request('/api/terminals','POST',{}, {'X-Workspace-Token':'wrong'})).status,403); + const rebound = await new Promise(resolve => http.get(app.origin + '/api/status', {headers:{Host:'evil.example'}}, res => { res.resume(); resolve(res.statusCode); })); + assert.equal(rebound,403); + assert.equal((await request('/api/status','GET',undefined, {'Sec-Fetch-Site':'cross-site'})).status,403); + assert.equal((await request('/api/terminals','POST',{input:'x'.repeat(17000)})).status,400); + const {id} = await (await request('/api/terminals','POST',{})).json(); + const input = value => request(`/api/terminals/${id}/input`,'POST',{input:value}); + let output = '', offset = 0; + const until = async pattern => { + for (let attempt = 0; attempt < 100; attempt++) { + const result = await (await request(`/api/terminals/${id}/output?after=${offset}`)).json(); + output += result.output; offset = result.offset; + if (pattern.test(output)) return; + await new Promise(resolve => setTimeout(resolve, 30)); + } + assert.fail(`Expected ${pattern}: ${output}`); + }; + await until(/workspace \$/); + await input("printf 'proof' > proof.txt\nexport DEMO_PERSIST=held\n"); + await input("printf '\\nSTATE:%s\\n' \"$DEMO_PERSIST\"\npwd\n"); + await until(/STATE:held/); + assert.equal(await fs.readFile(path.join(workspace,'proof.txt'),'utf8'),'proof'); + await input('sleep 30\n'); + await new Promise(resolve => setTimeout(resolve,100)); + await input('\u0003'); + await input("printf '\\nINTERRUPTED_OK\\n'\n"); + await until(/\r?\nINTERRUPTED_OK\r?\n/); + assert.equal((await request(`/api/terminals/${id}`,'DELETE',{})).status,200); + assert.equal((await request(`/api/terminals/${id}/output`)).status,404); + assert.equal((await request('/server.mjs')).status,404); + } finally { await app.close(); await fs.rm(workspace,{recursive:true,force:true}); } +}); diff --git a/portable/workspace/terminal_bridge.py b/portable/workspace/terminal_bridge.py new file mode 100644 index 0000000..92f9734 --- /dev/null +++ b/portable/workspace/terminal_bridge.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""One PTY per helper; JSON lines in, raw terminal bytes out. No sandbox.""" +import fcntl +import json +import os +import pty +import select +import signal +import struct +import sys +import termios + +pid, master = pty.fork() +if pid == 0: + os.chdir(sys.argv[1]) + os.execve('/bin/bash', ['bash', '--noprofile', '--norc', '-i'], { + 'PATH': '/usr/local/bin:/usr/bin:/bin', 'HOME': sys.argv[1], + 'TERM': 'dumb', 'PS1': 'workspace $ ', 'LANG': 'C.UTF-8', + }) + +fcntl.ioctl(master, termios.TIOCSWINSZ, struct.pack('HHHH', 24, 100, 0, 0)) +pending = b'' +try: + while True: + readable, _, _ = select.select([master, sys.stdin.buffer], [], []) + if master in readable: + try: + data = os.read(master, 65536) + except OSError: + break + if not data: + break + sys.stdout.buffer.write(data) + sys.stdout.buffer.flush() + if sys.stdin.buffer in readable: + data = os.read(sys.stdin.fileno(), 65536) + if not data: + break + pending += data + while b'\n' in pending: + line, pending = pending.split(b'\n', 1) + command = json.loads(line) + data = command['input'].encode('utf-8') + while data: + count = os.write(master, data) + data = data[count:] +finally: + try: + foreground = os.tcgetpgrp(master) + if foreground != pid: + os.killpg(foreground, signal.SIGHUP) + except (OSError, ProcessLookupError): + pass + os.close(master) + try: + os.killpg(pid, signal.SIGHUP) + except ProcessLookupError: + pass + os.waitpid(pid, 0) diff --git a/portable/workspace/ui/README.md b/portable/workspace/ui/README.md new file mode 100644 index 0000000..c754cc2 --- /dev/null +++ b/portable/workspace/ui/README.md @@ -0,0 +1,96 @@ +# Workspace interface + +Dependency-free HTML, CSS and JavaScript for a portable agent workspace. Serve +this directory through the sibling workspace server +([`../server.mjs`](../server.mjs)) for a real local shell and the real agent +lifecycle adapter. Opening it through a plain static server still exercises the +interface: the agent composer then reports that no workspace server is +answering instead of fabricating output. No external fonts, scripts or image +services are loaded. + +The agent tab is wired to the server’s real agent lifecycle API. The backend +embeds the vendored Pi SDK with a **scripted local model** — model turns are a +local script, while the admitted `read`/`write` tools execute for real inside +the server workspace. The interface labels this honestly: “Local scripted +agent; real tools”. It is not provider inference. When the SDK is not built, +`GET /api/status` reports the reason and the composer shows it; no canned +success is ever displayed in its place. Tor connection and funding states +remain explicitly simulated fixtures until live adapters exist. + +Keyboard access: Tab navigates controls, Ctrl/Cmd+K focuses the agent composer, +Ctrl/Cmd+Enter submits, and arrow keys switch between terminal/file tabs and +between terminal session tabs. Ctrl+C in the command input interrupts the +selected shell. Below 1101px (with the sidebar visible) the left navigation +switches between the full agent and terminal panes; wider screens keep the +split view, and the stacked single-column layout below 651px is unchanged. +The shell view is a plain-text PTY transcript, not a full-screen terminal +emulator. + +## API use + +All requests use the same origin. Mutation requests send Content-Type +`application/json` and `X-Workspace-Token` from the status response. + +- `GET /api/status`: `{terminalAvailable, csrfToken, workspace, agent}`. The + composer hint and submit gating come from `agent {available, kind, detail}`; + unavailable runs surface the server’s reason plus how to fix it. +- `POST /api/agents` `{prompt}`: `201 {id}` starts a run. `503` shows the + unavailable detail, `409` reports capacity; in every failure case the prompt + stays in the composer and nothing is resent automatically. The prompt is only + cleared after a `201`. +- `GET /api/agents/:id/events?after=N`: `{events, offset, truncated, state}`. + The UI polls with the previous offset (450 ms) and renders streamed + `text_delta` text as it arrives, tool progress lines (`tool_start`/`tool_end` + with the target path), and terminal outcomes (`done` reports that the scripted + agent wrote `agent-run.txt` in the server workspace — inspectable from the + terminal tab; the UI has no file-read API and does not pretend to show it). + `truncated` is surfaced instead of hidden. Poll failures retry a bounded + number of times, then report a lost stream without resubmitting anything. + Polling continues until the `state` field is terminal. The payload-free + `worker_reported_done` marker records a provisional worker claim. The + backend publishes canonical `done` and its completion details only after + the worker process actually exited cleanly, so a done-then-hang or + done-then-nonzero worker ends as a still-polled run settling to + `timeout`/`error`, never as success. +- `DELETE /api/agents/:id`: Stop awaits the backend outcome (the event poll + must reach a terminal state) with an abort timeout; a timed-out stop says so + instead of claiming success. +- `GET /api/terminals`: `{sessions: [{id, exited}]}`. Terminal sessions are + tabs: “New session” opens an additional PTY and closes nothing, and the + explicit “Close session” action closes only the selected PTY. Each tab + retains its own transcript buffer, output offset and command draft, so + switching reattaches exactly that session; only the selected session is + polled, a switch cancels the previous poll loop (a late poll or input + result is never applied to a different terminal), and Ctrl+C targets the + selected session. On load the UI restores this tab’s surviving sessions + from sessionStorage — including which session was selected — replaying each + transcript from the server’s bounded buffer; sessions that no longer exist + are reported, sessions from other tabs are never adopted, and no automatic + replacement is created. The server’s eight-session cap surfaces as a + message when “New session” hits it, leaving existing sessions untouched. +- Terminal output polling (`after=` previous offset), input sending and Ctrl+C + behave as before; input mutations are never silently retried. + +Reload behavior: this tab’s terminal session list and selected session, the +agent run id and the composer draft are kept in `sessionStorage`, so a reload +restores the same PTYs (selected tab included, transcripts replayed) and +reattaches the same agent run (from event offset 0) instead of restarting +work. New tabs start fresh sessions. + +## Preview fixture, visibly separate + +Connection/credit scenarios and the browser “hello.txt” preview remain as +explicit fixtures under “Preview scenarios” in the footer. The preview fixture +runs the old scripted interaction entirely in browser memory, labelled +“Preview · browser fixture”, and never contacts the workspace server. The +file tab shows that browser-memory fixture only; real agent writes land in the +server workspace and are inspected through the terminal. + +## Rendering policy + +All server- and model-derived text is rendered with `textContent`/text nodes; +`innerHTML` is never used. Terminal control sequences are stripped, never +executed. A live provider adapter must keep these properties: honest +unavailable states, no automatic replay of ambiguous starts or inputs, and +connection/funding labels that never upgrade to live claims without +end-to-end evidence. diff --git a/portable/workspace/ui/app.js b/portable/workspace/ui/app.js new file mode 100644 index 0000000..2d23d6c --- /dev/null +++ b/portable/workspace/ui/app.js @@ -0,0 +1,627 @@ +const $ = (id) => document.getElementById(id); + +// Honest labels: the backend agent is a local scripted model driving real SDK +// tools. Connection and funding states stay simulated fixtures. +const AGENT_LABEL = 'Local scripted agent; real tools'; +const RUN_LABELS = { + starting: 'Starting', running: 'Working', done: 'Complete', aborted: 'Stopped', + error: 'Error', timeout: 'Timed out', killed: 'Terminated', unavailable: 'Unavailable', + gone: 'Ended (not on server)', +}; +const TERMINAL_RUN_STATES = ['done', 'aborted', 'error', 'timeout', 'killed', 'unavailable']; +const AGENT_POLL_MS = 450; +const AGENT_STOP_TIMEOUT_MS = 8000; +const MAX_POLL_FAILURES = 6; + +const store = { + get(key) { try { return sessionStorage.getItem(key); } catch { return null; } }, + set(key, value) { try { sessionStorage.setItem(key, value); } catch { /* storage unavailable */ } }, + remove(key) { try { sessionStorage.removeItem(key); } catch { /* storage unavailable */ } }, +}; + +const state = { + scenario: 'funded', payment: 0, run: 0, running: false, runningPreview: false, + token: null, terminalBusy: false, + serverOk: null, agent: { available: false, kind: null, detail: null }, + agentRun: null, pollFailures: 0, + // One entry per retained PTY session: {id, label, buffer, offset, exited, + // disconnected, pollToken, draft, note}. Only the selected session is + // polled; switching reattaches from that session's own offset. + terminalSessions: [], selectedTerminal: null, +}; +let streamText = null; // active streaming assistant text node +let streamedChars = 0; // whether any text_delta arrived this run +const pendingTools = []; + +const phases = ['Awaiting payment', 'Payment detected', 'Confirming', 'Converting', 'Credit ready']; +const connected = () => ['funded', 'unfunded'].includes(state.scenario); + +function render() { + const labels = { funded: 'Tor ready · simulated', unfunded: 'Tor ready · simulated', connecting: 'Tor connecting · simulated', offline: 'Offline · simulated', reconnect: 'Tor interrupted · simulated' }; + $('connection').textContent = labels[state.scenario]; + $('connection').classList.toggle('waiting', !connected()); + $('credit').textContent = state.scenario === 'funded' ? '$4.82 credit ↗' : state.scenario === 'unfunded' ? 'Add credit ↗' : 'Credit unknown ↗'; + $('credit-freshness').textContent = state.scenario === 'funded' ? 'Example provider · fresh fixture' : state.scenario === 'unfunded' ? 'Example provider · $0 fixture' : 'No balance refresh'; + $('payment-steps').hidden = !connected(); + $('credit').title = state.scenario === 'funded' ? 'Example provider · simulated balance · refreshed just now' : 'No live balance has been fetched'; + $('composer-hint').textContent = agentHint(); + $('funding-description').textContent = connected() ? 'Example provider credit. This preview demonstrates the funding journey without a wallet or payment address.' : 'Waiting for a connection. Your balance is unknown; no payment instruction is available.'; + $('payment-steps').replaceChildren(...phases.map((phase, i) => { const item = document.createElement('li'); item.textContent = phase; item.className = i <= state.payment ? 'done' : ''; if (i === state.payment) item.setAttribute('aria-current', 'step'); return item; })); + $('advance-payment').textContent = state.payment === 4 ? 'Reset funding preview' : `Simulate: ${phases[state.payment + 1].toLowerCase()}`; + $('advance-payment').disabled = !connected(); +} + +function agentHint() { + if (state.serverOk === null) return 'Checking workspace server…'; + if (state.serverOk === false) return 'No workspace server · static preview'; + if (!state.agent.available) return `Agent unavailable · ${state.agent.detail || 'Pi SDK not built'}`; + if (state.agent.kind === 'fixture') return 'Fixture transport (server tests only)'; + return AGENT_LABEL; +} + +function unavailableMessage() { + if (state.serverOk !== true) { + return 'No workspace server is answering /api/status. Start one with: node portable/workspace/server.mjs --workspace (see portable/workspace/README.md), then reload this page. This interface will not fabricate agent output.'; + } + const detail = state.agent.detail || 'the vendored Pi SDK is not built'; + return `Agent unavailable: ${detail}. Build the vendored SDK and restart the workspace server (see the “Agent adapter” section of portable/workspace/README.md), then reload. Real output is never faked; terminals keep working and the preview fixture in the footer can still demonstrate the interface.`; +} + +function appendActivity(el) { $('activity').append(el); $('activity').scrollTop = $('activity').scrollHeight; } +function message(kind, text, labelText) { + $('activity').querySelector('.welcome')?.remove(); + const el = document.createElement('div'); el.className = `message ${kind}`; + const label = document.createElement('strong'); + label.textContent = labelText ?? (kind === 'user' ? 'You' : kind === 'preview' ? 'Preview · browser fixture' : kind === 'agent' ? 'Agent · scripted local' : 'Status'); + el.append(label, document.createTextNode(text)); appendActivity(el); +} +function toolLine(name, detail) { + const el = document.createElement('div'); el.className = 'message tool'; + const label = document.createElement('strong'); label.textContent = `Tool · ${name}`; + el.append(label, document.createTextNode(` → ${detail}`)); + el.dataset.tool = name; appendActivity(el); return el; +} +function toolDetail(args) { + if (typeof args !== 'string' || !args) return '(no arguments shown)'; + try { const parsed = JSON.parse(args); if (parsed && typeof parsed.path === 'string') return parsed.path; } catch { /* raw JSON string */ } + return args.length > 220 ? `${args.slice(0, 220)}…` : args; +} +function openStream() { + if (streamText) return; + $('activity').querySelector('.welcome')?.remove(); + const el = document.createElement('div'); el.className = 'message agent'; + const label = document.createElement('strong'); label.textContent = 'Agent · scripted local'; + streamText = document.createTextNode(''); + el.append(label, streamText); appendActivity(el); +} + +// ---- Agent lifecycle (real backend adapter) ---- + +function isTerminalRun(runState) { return TERMINAL_RUN_STATES.includes(runState); } +function runLabel(runState) { return RUN_LABELS[runState] ?? runState; } + +function renderAgentEvent(event) { + if (event.type === 'state' && event.state === 'running') { $('agent-status').textContent = RUN_LABELS.running; return; } + if (event.type === 'agent_event') { + const inner = event.event; + if (inner === 'message_start') { if (event.role === 'assistant') openStream(); return; } + if (inner === 'text_delta') { openStream(); streamText.appendData(String(event.delta ?? '')); streamedChars += String(event.delta ?? '').length; return; } + if (inner === 'message_end') { streamText = null; return; } + if (inner === 'tool_start') { pendingTools.push(toolLine(String(event.tool ?? 'tool'), toolDetail(event.args))); return; } + if (inner === 'tool_end') { + const name = String(event.tool ?? 'tool'); + const index = pendingTools.findIndex(el => el.dataset.tool === name); + if (index !== -1) { const el = pendingTools[index]; el.append(document.createTextNode(event.isError ? ' · failed' : ' · done')); pendingTools.splice(index, 1); } + return; + } + if (inner === 'auto_retry_start') { message('system', `SDK retry started (attempt ${event.attempt ?? '?'}); retries are supposed to be disabled.`); return; } + return; // lifecycle markers (agent_start, turn markers) stay visible via status + } + if (event.type === 'worker_reported_done' || event.type === 'done') { + // Worker completion is provisional until the manager confirms clean exit. + if (state.agentRun) state.agentRun.reportedOutcome = event; + return; + } + if (event.type === 'aborted') { if (state.agentRun) state.agentRun.sawOutcome = true; message('system', 'Run stopped. Your draft and workspace files remain.'); return; } + if (event.type === 'unavailable') { if (state.agentRun) state.agentRun.sawOutcome = true; message('system', `Agent became unavailable: ${event.reason ?? 'unknown reason'}.`); return; } + if (event.type === 'error' || event.type === 'fatal') { + if (state.agentRun) state.agentRun.sawOutcome = true; + message('system', `Run failed: ${event.message ?? event.error ?? event.reason ?? 'unknown error'}. Nothing is resent automatically.`); + } +} + +function resetRunTranscriptState() { streamText = null; streamedChars = 0; pendingTools.length = 0; state.pollFailures = 0; } + +function setRunningUI(running) { + $('stop-agent').hidden = !running; + $('stop-agent').disabled = false; + $('send-agent').disabled = running; +} + +function finishRun(runState) { + const run = state.agentRun; + if (!run || run.ended) return; + run.ended = true; state.running = false; + setRunningUI(false); + $('agent-status').textContent = runLabel(runState); + if (runState === 'done') { + const outcome = run.reportedOutcome; + if (typeof outcome?.lastText === 'string' && outcome.lastText && streamedChars === 0) message('agent', outcome.lastText); + const wrote = outcome?.wrote; + if (typeof wrote === 'string' && wrote) { + message('system', 'Run complete. The scripted agent wrote ' + wrote + ' in the server workspace through the real SDK write tool. Inspect the file from the Terminal tab. Scripted model, not provider inference.'); + } else { + message('system', 'Run complete. The local scripted task finished; no file write was reported. Scripted model, not provider inference.'); + } + return; + } + // The terminal state can be observed before the worker's outcome event line + // lands, so the status message must not depend on that event arriving. + if (runState === 'gone') message('system', 'The server no longer reports this run; it finished, was stopped elsewhere, or the server restarted.'); + else if (runState === 'timeout' || runState === 'killed') message('system', `The server ended this run (${runLabel(runState).toLowerCase()}).`); + else if (!run.sawOutcome) message('system', run.stopAcked ? 'Run stopped. Your draft and workspace files remain.' : `Run ended (${runLabel(runState).toLowerCase()}). Nothing was resent automatically.`); +} + +async function pollAgent() { + const run = state.agentRun; + if (!run || run.ended) return; + try { + const result = await api(`agents/${encodeURIComponent(run.id)}/events?after=${run.offset}`); + if (state.agentRun !== run) return; + state.pollFailures = 0; + run.offset = result.offset; + for (const event of result.events) renderAgentEvent(event); + if (result.truncated) message('system', 'Older activity was dropped by the server’s bounded event buffer.'); + if (isTerminalRun(result.state)) { finishRun(result.state); return; } + setTimeout(pollAgent, AGENT_POLL_MS); + } catch (error) { + if (state.agentRun !== run) return; + if (error.status === 404) { finishRun(run.stopAcked ? 'aborted' : 'gone'); return; } + state.pollFailures += 1; + if (state.pollFailures >= MAX_POLL_FAILURES) { + $('agent-status').textContent = 'Stream lost'; + message('system', `Lost the agent event stream after ${state.pollFailures} tries (${error.message}). The run may still be active on the server; nothing is resubmitted automatically. Reload this page to reattach.`); + run.ended = true; state.running = false; setRunningUI(false); + return; + } + $('agent-status').textContent = 'Reconnecting…'; + setTimeout(pollAgent, 1000); + } +} + +async function startAgent(prompt) { + if (state.serverOk === true && !state.agent.available) { message('system', unavailableMessage()); return; } + state.running = true; setRunningUI(true); + $('stop-agent').disabled = true; // no run id yet: Stop cannot act until the 201 returns + $('agent-status').textContent = RUN_LABELS.starting; + try { + const result = await api('agents', 'POST', { prompt }); + $('prompt').value = ''; store.remove('portable-prompt'); + resetRunTranscriptState(); + state.agentRun = { id: result.id, offset: 0, ended: false }; + store.set('portable-agent', JSON.stringify({ id: result.id })); + $('stop-agent').disabled = false; + message('system', `Started run ${result.id}. ${AGENT_LABEL}.`); + pollAgent(); + } catch (error) { + state.running = false; setRunningUI(false); + $('agent-status').textContent = 'Not started'; + if (error.status === 503) { + state.agent.available = false; state.agent.detail = error.detail || state.agent.detail || 'Pi SDK unavailable'; + render(); message('system', unavailableMessage()); + } else if (error.status === 409) { + message('system', `Agent capacity is reached (${error.detail || 'too many active runs'}). The prompt stays in the composer; try again once a run finishes.`); + } else if (error instanceof TypeError) { + // Unknown outcome: no response arrived, so the run may or may not have + // started. A run existing on the server is not proof it is this prompt + // (it may be another tab's), so nothing is attached or started here. + message('system', 'The start request could not reach the server, so its outcome is unknown. The prompt stays in the composer; it is not resent automatically.'); + try { + const listing = await api('agents'); + const alive = (listing.agents ?? []).filter(agent => agent.alive); + if (alive.length === 0) message('system', 'The server lists no active runs, so the prompt most likely never started.'); + else message('system', `The server lists ${alive.length} active run${alive.length === 1 ? '' : 's'} (${alive.map(agent => agent.id).join(', ')}). An existing run is not evidence this prompt started — it may be another tab’s — so nothing is attached automatically. Reconcile explicitly: inspect the runs from the terminal, or reload to reattach a run this tab saved.`); + } catch { /* listing unavailable too; the outcome stays unknown */ } + } else { + message('system', `${error.message}. The prompt is still in the composer; it was not resent automatically.`); + } + } +} + +async function stopAgent() { + const run = state.agentRun; + if (!run || run.ended) return; + $('stop-agent').disabled = true; + $('agent-status').textContent = 'Stopping…'; + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), AGENT_STOP_TIMEOUT_MS); + try { + // The DELETE resolves only after the worker has exited (or the kill + // grace elapsed), so awaiting it is awaiting the backend outcome. + await api(`agents/${encodeURIComponent(run.id)}`, 'DELETE', undefined, controller.signal); + run.stopAcked = true; + let outcome = 'aborted'; + try { + // Drain any final events (abort line, streamed text) before concluding; + // the record is reaped after a stop, so a 404 here is expected. + const result = await api(`agents/${encodeURIComponent(run.id)}/events?after=${run.offset}`); + if (state.agentRun === run) { run.offset = result.offset; for (const event of result.events) renderAgentEvent(event); if (isTerminalRun(result.state)) outcome = result.state; } + } catch { /* record already reaped by the confirmed stop */ } + if (state.agentRun === run) finishRun(outcome); + } catch (error) { + if (error.name === 'AbortError') message('system', 'The stop request timed out. Still waiting for the server’s outcome; the run is not assumed stopped and nothing is resent.'); + else if (error.status === 404) { run.stopAcked = true; if (state.agentRun === run) finishRun('gone'); } + else message('system', `Stop request failed: ${error.message}. Still waiting for the run’s outcome via the event stream.`); + if (state.agentRun === run && !run.ended) $('stop-agent').disabled = false; + } finally { + clearTimeout(timer); + } +} + +async function initAgentReattach() { + const saved = store.get('portable-agent'); + if (!saved) return; + let id = null; + try { id = JSON.parse(saved).id; } catch { /* corrupted entry */ } + if (!id) { store.remove('portable-agent'); return; } + try { + const result = await api(`agents/${encodeURIComponent(id)}/events?after=0`); + resetRunTranscriptState(); + state.agentRun = { id, offset: result.offset, ended: false }; + for (const event of result.events) renderAgentEvent(event); + if (isTerminalRun(result.state)) { + finishRun(result.state); + if (result.events.length === 0) message('system', `Reloaded; run ${id} already ${runLabel(result.state).toLowerCase()}.`); + } else { + state.running = true; setRunningUI(true); + $('agent-status').textContent = `Reattached · ${runLabel(result.state).toLowerCase()}`; + pollAgent(); + } + } catch (error) { + if (error.status === 404) { store.remove('portable-agent'); } + else message('system', `Could not reattach to run ${id} (${error.message}). Nothing is restarted automatically.`); + } +} + +// ---- Explicit browser preview fixture (no backend involvement) ---- + +async function runPreviewFixture() { + if (state.running) { message('preview', 'A real agent run is active; the preview fixture waits until it finishes.'); return; } + if (state.runningPreview) return; + if (!connected()) { message('preview', 'The preview fixture needs a connected scenario (see Connection & credit below).'); return; } + if (state.scenario === 'unfunded') { + $('funding').hidden = false; $('credit').setAttribute('aria-expanded', 'true'); + message('preview', 'This scenario needs compute credit. Use the simulated funding steps; no real payment is involved.'); + return; + } + state.runningPreview = true; $('preview-agent').disabled = true; + message('preview', 'Running the scripted preview entirely in this browser. No workspace server, model or tools are involved.'); + await new Promise(resolve => setTimeout(resolve, 1200)); + $('file-content').textContent = 'Hello, world.\n\nA little computer. A place for your ideas.\n\nCreated by the preview fixture, in browser memory only.\n'; + message('preview', 'Created hello.txt in the browser-memory file viewer — a fixture, not the server workspace and not a real write.'); + state.runningPreview = false; $('preview-agent').disabled = false; +} + +// ---- Shared API helper (textContent-only rendering of anything it feeds) ---- + +async function api(path, method = 'GET', body, signal) { + const response = await fetch(`/api/${path}`, { + method, + headers: { 'Content-Type': 'application/json', ...(state.token ? { 'X-Workspace-Token': state.token } : {}) }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + ...(signal ? { signal } : {}), + }); + if (!response.ok) { + let detail = ''; + try { const data = await response.json(); detail = data?.detail ?? data?.error ?? ''; } catch { /* non-JSON error body */ } + const error = new Error(`Request failed (${response.status})`); + error.status = response.status; error.detail = typeof detail === 'string' ? detail : ''; + throw error; + } + return response.json(); +} + +function applyStatus(status) { + state.token = status.csrfToken ?? state.token; + state.serverOk = true; + if (status.agent && typeof status.agent === 'object') { + state.agent = { available: status.agent.available === true, kind: status.agent.kind ?? null, detail: status.agent.detail ?? null }; + } + if (!state.agentRun) $('agent-status').textContent = state.agent.available ? 'Ready' : 'Unavailable'; + render(); +} + +// ---- Terminal: retained PTY sessions with visible tabs ---- +// "New session" opens an additional PTY and closes nothing; only the explicit +// Close action closes a session, and it closes just the selected one. Each +// session retains its own transcript buffer, output offset and command draft, +// so switching tabs reattaches exactly that session's state. Only the +// selected session is polled; switching bumps the poll token so a late poll +// or input result is never applied to a different terminal. + +// Plain text PTY view: strip control sequences rather than execute terminal escape codes. +function plainTerminal(value) { return value.replace(/\x1b\][^\x07]*(?:\x07|\x1b\\)/g, '').replace(/\x1b\[[0-?]*[ -/]*[@-~]/g, '').replace(/\r\n/g, '\n').replace(/[\x00-\x08\x0b-\x1f\x7f]/g, ''); } +const TERMINAL_NOTE = 'Real commands run on this development host. This terminal has no demonstrated Tor or isolation boundary. Ctrl+C interrupts; full-screen terminal apps are not supported.'; +const REATTACH_NOTE = 'Reattached to this tab’s surviving shell after the reload; no new session was created. Switching tabs replays each session’s retained server transcript.'; + +function findSession(id) { return state.terminalSessions.find(session => session.id === id); } + +function storeTerminals() { + store.set('portable-terminals', JSON.stringify({ + list: state.terminalSessions.map(session => ({ id: session.id, label: session.label })), + selected: state.selectedTerminal, + initialized: true, + })); +} +function loadTerminals() { + const raw = store.get('portable-terminals'); + if (raw) { + try { + const saved = JSON.parse(raw); + const list = Array.isArray(saved?.list) + ? saved.list.filter(entry => entry && typeof entry.id === 'string' && typeof entry.label === 'string').map(entry => ({ id: entry.id, label: entry.label })) + : []; + return { list, selected: typeof saved?.selected === 'string' ? saved.selected : null, initialized: saved?.initialized === true || list.length > 0 }; + } catch { /* corrupted entry: fall through to a fresh start */ } + } + const legacy = store.get('portable-terminal'); + if (legacy) { store.remove('portable-terminal'); return { list: [{ id: legacy, label: '1' }], selected: legacy, initialized: true }; } + return { list: [], selected: null, initialized: false }; +} + +function renderTerminalOutput(session) { + $('terminal-output').textContent = session.buffer; + $('terminal-output').scrollTop = $('terminal-output').scrollHeight; +} +function appendSessionOutput(session, text) { + session.buffer = (session.buffer + plainTerminal(text)).slice(-100000); + if (session.id === state.selectedTerminal) renderTerminalOutput(session); +} + +function renderTerminalTabs() { + $('terminal-tabs').replaceChildren(...state.terminalSessions.map(session => { + const tab = document.createElement('button'); + tab.id = `terminal-session-${session.label}`; + tab.className = 'terminal-session-tab'; + tab.setAttribute('role', 'tab'); + tab.setAttribute('aria-selected', String(session.id === state.selectedTerminal)); + tab.setAttribute('aria-controls', 'terminal-output'); + tab.tabIndex = session.id === state.selectedTerminal ? 0 : -1; + tab.textContent = `Shell ${session.label}${session.exited ? ' · exited' : ''}`; + tab.addEventListener('click', () => selectTerminal(session.id)); + tab.addEventListener('keydown', event => { + if (!['ArrowLeft', 'ArrowRight', 'Home', 'End'].includes(event.key)) return; + event.preventDefault(); + const ids = state.terminalSessions.map(candidate => candidate.id); + const index = ids.indexOf(session.id); + const next = event.key === 'Home' ? 0 : event.key === 'End' ? ids.length - 1 + : (index + (event.key === 'ArrowRight' ? 1 : ids.length - 1)) % ids.length; + selectTerminal(ids[next]); + $(`terminal-session-${state.terminalSessions[next].label}`).focus(); + }); + return tab; + })); +} + +function renderTerminalView() { + renderTerminalTabs(); + const session = findSession(state.selectedTerminal); + $('close-terminal').disabled = !session || state.terminalBusy; + if (!session) { + $('terminal-output').textContent = ''; + $('terminal-state').textContent = 'No terminal sessions'; + $('terminal-note').textContent = 'This tab has no open terminal. New session opens another PTY; sessions from other tabs are not adopted automatically.'; + $('command').value = ''; + $('command').disabled = true; + return; + } + renderTerminalOutput(session); + $('terminal-state').textContent = session.exited ? 'Shell exited' : session.disconnected ? 'Terminal disconnected' : 'Live local shell'; + $('terminal-note').textContent = session.note ?? TERMINAL_NOTE; + $('command').value = session.draft ?? ''; + $('command').disabled = session.exited || session.disconnected; +} + +async function pollSession(session) { + const token = ++session.pollToken; + while (!session.exited && !session.disconnected && session.pollToken === token && state.selectedTerminal === session.id) { + try { + const result = await api(`terminals/${encodeURIComponent(session.id)}/output?after=${session.offset}`); + if (session.pollToken !== token) return; // a newer loop owns this session now + session.offset = result.offset; + if (result.truncated) session.note = 'Older terminal output was discarded by the server’s bounded buffer.'; + if (result.output) appendSessionOutput(session, result.output); + if (result.exited) { + session.exited = true; + session.note = 'The shell process exited. This tab keeps its transcript; commands cannot be sent to it.'; + renderTerminalView(); + return; + } + } catch (error) { + if (session.pollToken !== token) return; + session.disconnected = true; + session.note = `${error.message}. Polling stopped for this session; selecting its tab again retries once. Commands are never resent.`; + renderTerminalView(); + return; + } + await new Promise(resolve => setTimeout(resolve, 350)); + } +} + +function selectTerminal(id) { + const session = findSession(id); + if (!session) return; + state.selectedTerminal = id; + storeTerminals(); + if (session.disconnected) { session.disconnected = false; session.note = TERMINAL_NOTE; } // explicit reselect retries once + renderTerminalView(); + if (!session.exited) pollSession(session); +} + +function addSession(id, note) { + const used = state.terminalSessions.reduce((max, session) => Math.max(max, Number(session.label) || 0), 0); + const session = { id, label: String(used + 1), buffer: '', offset: 0, exited: false, disconnected: false, pollToken: 0, draft: '', note }; + state.terminalSessions.push(session); + selectTerminal(session.id); + return session; +} + +async function initTerminal() { + try { + const status = await api('status'); + applyStatus(status); + if (!status.terminalAvailable) throw new Error('Terminal service is unavailable'); + const listing = await api('terminals'); + const alive = new Set((listing.sessions ?? []).filter(session => !session.exited).map(session => session.id)); + const saved = loadTerminals(); + const restored = saved.list.filter(entry => alive.has(entry.id)).map(entry => ({ + id: entry.id, label: entry.label, buffer: '', offset: 0, exited: false, disconnected: false, pollToken: 0, draft: '', note: REATTACH_NOTE, + })); + const dropped = saved.list.length - restored.length; + if (restored.length) { + state.terminalSessions = restored; + if (dropped > 0) { + const target = restored.find(session => session.id === saved.selected) ?? restored[restored.length - 1]; + target.note = `${REATTACH_NOTE} ${dropped} previously open session${dropped === 1 ? '' : 's'} no longer exist${dropped === 1 ? 's' : ''} on the server.`; + } + selectTerminal(restored.some(session => session.id === saved.selected) ? saved.selected : restored[restored.length - 1].id); + } else if (saved.initialized) { + // This tab opened terminals before and none survived (closed here, or + // the server restarted): create nothing and adopt nothing. + state.selectedTerminal = null; + renderTerminalView(); + if (dropped > 0) $('terminal-note').textContent = `${dropped} previously open session${dropped === 1 ? '' : 's'} from this tab no longer exist${dropped === 1 ? 's' : ''} on the server.`; + } else { + const result = await api('terminals', 'POST', {}); + addSession(result.id, TERMINAL_NOTE); + } + } catch (error) { + if (error instanceof TypeError) state.serverOk = false; + render(); + state.selectedTerminal = null; + renderTerminalTabs(); + $('terminal-output').textContent = `${error.message}\n`; + $('terminal-state').textContent = 'Terminal unavailable'; + $('terminal-note').textContent = 'Start the local workspace server to use a real shell. This static preview does not simulate command execution.'; + $('command').disabled = true; + $('close-terminal').disabled = true; + } +} + +async function newTerminal() { + if (state.terminalBusy) return; + state.terminalBusy = true; + $('new-terminal').disabled = true; + try { + if (!state.token) applyStatus(await api('status')); + const result = await api('terminals', 'POST', {}); + addSession(result.id, TERMINAL_NOTE); // adds a PTY; every existing session stays open + } catch (error) { + const reason = error.status === 409 + ? `Terminal capacity reached (${error.detail || 'too many sessions on this server'}). Close a session before opening another; open sessions keep running.` + : `${error instanceof TypeError ? 'Workspace server unreachable' : error.message}. No session was created.`; + const session = findSession(state.selectedTerminal); + if (session) appendSessionOutput(session, `\n${reason}\n`); + else { $('terminal-state').textContent = 'Terminal unavailable'; $('terminal-note').textContent = reason; } + } finally { + state.terminalBusy = false; + $('new-terminal').disabled = false; + $('close-terminal').disabled = !findSession(state.selectedTerminal); + } +} + +async function closeTerminal() { + const session = findSession(state.selectedTerminal); + if (!session || state.terminalBusy) return; + state.terminalBusy = true; + $('close-terminal').disabled = true; + $('new-terminal').disabled = true; + session.pollToken += 1; // stop this session's poll loop before closing it + try { + await api(`terminals/${encodeURIComponent(session.id)}`, 'DELETE'); + } catch (error) { + if (error.status !== 404) { + appendSessionOutput(session, `\n${error.message}. The session was not closed; it stays selected.`); + state.terminalBusy = false; + $('new-terminal').disabled = false; + renderTerminalView(); + if (!session.exited && state.selectedTerminal === session.id) { session.disconnected = false; pollSession(session); } + return; + } // 404: the server no longer knows it; drop the tab + } + const index = state.terminalSessions.indexOf(session); + state.terminalSessions.splice(index, 1); + const next = state.terminalSessions[index] ?? state.terminalSessions[index - 1]; + state.terminalBusy = false; + $('new-terminal').disabled = false; + if (next) selectTerminal(next.id); + else { state.selectedTerminal = null; storeTerminals(); renderTerminalView(); } +} + +// ---- Wiring ---- + +function switchTool(view) { + for (const name of ['terminal', 'file']) { const active = name === view; $(`${name}-tab`).setAttribute('aria-selected', String(active)); $(`${name}-tab`).tabIndex = active ? 0 : -1; $(`${name}-view`).hidden = !active; } +} +// Narrow screens (651–1100px) show one workspace pane at a time and the +// sidebar switches between them; wider screens keep the split view and +// narrower-than-651px screens keep the stacked single-column layout. +function setPane(pane) { + document.body.dataset.pane = pane; + $('agent-nav').classList.toggle('selected', pane === 'agent'); + $('terminal-nav').classList.toggle('selected', pane === 'tools'); +} +$('credit').addEventListener('click', () => { const open = $('funding').hidden; $('funding').hidden = !open; $('credit').setAttribute('aria-expanded', String(open)); }); +$('advance-payment').addEventListener('click', () => { state.payment = (state.payment + 1) % phases.length; if (state.payment === 4) { state.scenario = 'funded'; $('scenario').value = 'funded'; message('preview', 'Funding preview complete. Example provider credit is ready. No real funds moved.'); } render(); }); +$('scenario').addEventListener('change', () => { state.scenario = $('scenario').value; if (state.scenario === 'unfunded') { state.payment = 0; $('funding').hidden = false; $('credit').setAttribute('aria-expanded', 'true'); } render(); }); +$('preview-agent').addEventListener('click', runPreviewFixture); +$('example').addEventListener('click', () => { $('prompt').value = 'Run the scripted task: write agent-run.txt into the workspace.'; $('prompt').focus(); }); +$('agent-form').addEventListener('submit', async (event) => { + event.preventDefault(); + const prompt = $('prompt').value.trim(); + if (!prompt || state.running || state.runningPreview) return; + message('user', prompt); + await startAgent(prompt); +}); +$('stop-agent').addEventListener('click', stopAgent); +$('prompt').addEventListener('input', () => store.set('portable-prompt', $('prompt').value)); +$('prompt').addEventListener('keydown', event => { if (event.key === 'Enter' && (event.ctrlKey || event.metaKey)) { event.preventDefault(); $('agent-form').requestSubmit(); } }); +document.addEventListener('keydown', event => { if ((event.ctrlKey || event.metaKey) && event.key.toLowerCase() === 'k') { event.preventDefault(); setPane('agent'); $('prompt').focus(); } }); +$('agent-nav').addEventListener('click', () => { setPane('agent'); $('prompt').focus(); }); +$('terminal-nav').addEventListener('click', () => { setPane('tools'); switchTool('terminal'); $('command').focus(); }); +$('file-nav').addEventListener('click', () => { setPane('tools'); switchTool('file'); $('file-tab').focus(); }); +for (const name of ['terminal', 'file']) { + $(`${name}-tab`).addEventListener('click', () => switchTool(name)); + $(`${name}-tab`).addEventListener('keydown', event => { if (['ArrowLeft', 'ArrowRight', 'Home', 'End'].includes(event.key)) { event.preventDefault(); const next = event.key === 'Home' ? 'terminal' : event.key === 'End' ? 'file' : name === 'terminal' ? 'file' : 'terminal'; switchTool(next); $(`${next}-tab`).focus(); } }); +} +$('new-terminal').addEventListener('click', newTerminal); +$('close-terminal').addEventListener('click', closeTerminal); +$('command').addEventListener('input', () => { const session = findSession(state.selectedTerminal); if (session) session.draft = $('command').value; }); +$('terminal-form').addEventListener('submit', async event => { + event.preventDefault(); + const session = findSession(state.selectedTerminal); + const input = $('command').value; + if (!session || session.exited || session.disconnected || !input.trim()) return; + try { + await api(`terminals/${encodeURIComponent(session.id)}/input`, 'POST', { input: `${input}\n` }); + // A late acknowledgement must not erase text typed since submission, + // nor the selected input of another retained shell. + if (session.draft === input) session.draft = ''; + if (state.selectedTerminal === session.id && $('command').value === input) $('command').value = ''; + } catch (error) { appendSessionOutput(session, `\n${error.message}. Input was not retried.\n`); } +}); +$('command').addEventListener('keydown', async event => { + if (!(event.ctrlKey && event.key.toLowerCase() === 'c')) return; + const session = findSession(state.selectedTerminal); // Ctrl+C targets the selected session only + if (!session || session.exited || session.disconnected) return; + event.preventDefault(); + try { await api(`terminals/${encodeURIComponent(session.id)}/input`, 'POST', { input: '\x03' }); } + catch (error) { appendSessionOutput(session, `\n${error.message}\n`); } +}); +setPane('agent'); + +const savedDraft = store.get('portable-prompt'); +if (savedDraft) $('prompt').value = savedDraft; +render(); +initTerminal(); +initAgentReattach(); diff --git a/portable/workspace/ui/index.html b/portable/workspace/ui/index.html new file mode 100644 index 0000000..751772d --- /dev/null +++ b/portable/workspace/ui/index.html @@ -0,0 +1,48 @@ + + + + + + + Your portable workspace + + + + + +
+ portable / workspace +
+
+
Local scripted agent; real tools
+
+ +
+
A LITTLE COMPUTER. YOUR NEXT BIG IDEA.

Make yourself at home.

workspace / 01
+ +
+
+

Your agent

Checking agent…
+

What shall we make?

A quiet place to think, build, and follow your curiosity.
Start with an idea. Open a terminal when you need one.

+
+
+
+
+
Checking terminal…

+ +
+
+ +
+
+ + diff --git a/portable/workspace/ui/style.css b/portable/workspace/ui/style.css new file mode 100644 index 0000000..789da96 --- /dev/null +++ b/portable/workspace/ui/style.css @@ -0,0 +1,11 @@ +:root{font-family:Inter,ui-sans-serif,system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;color:#e9e6dc;background:#191c1b;font-synthesis:none;--muted:#a3aaa4;--border:#363c37;--accent:#a6dfca;--panel:#202522}*{box-sizing:border-box}body{margin:0}button,input,textarea,select{font:inherit}button,a,input,textarea,select,summary{-webkit-tap-highlight-color:transparent}button{cursor:pointer;color:inherit}button:disabled{cursor:wait;opacity:.55}button:focus-visible,a:focus-visible,input:focus-visible,textarea:focus-visible,select:focus-visible,summary:focus-visible,pre:focus-visible{outline:2px solid var(--accent);outline-offset:4px}button{border:0}a{color:inherit;text-decoration:none}[hidden]{display:none!important}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);white-space:nowrap;border:0}.skip-link{position:fixed;left:12px;top:-80px;z-index:20;background:var(--accent);color:#172a21;padding:12px}.skip-link:focus{top:12px}.topbar{height:82px;padding:0 38px;display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid var(--border)}.wordmark{font-size:21px;font-weight:600;letter-spacing:-.6px}.wordmark-light{font-weight:400;color:var(--muted)}.mark{font-size:31px;vertical-align:-3px;color:var(--accent);margin-right:11px}.top-actions{display:flex;gap:25px;align-items:center}.status{font-size:12px;color:#c2c8c2;display:flex;align-items:center;gap:9px}.status:before,.fixture-dot{content:"";width:6px;height:6px;border-radius:50%;background:var(--accent);display:inline-block;flex-shrink:0}.status.waiting:before{background:#dcc08a}.credit{background:#2e3831;border:1px solid #4e6154;border-radius:7px;padding:9px 13px;font-size:13px}.fixture-banner{min-height:33px;display:flex;align-items:center;gap:9px;justify-content:center;background:#272a23;color:#d3c7a6;font-size:11px;padding:8px 15px}.fixture-dot{background:#d3c7a6;width:5px;height:5px}.banner-detail{color:#b2b19f}main{display:grid;grid-template-columns:213px 1fr;min-height:calc(100vh - 115px)}.sidebar{border-right:1px solid var(--border);padding:35px 18px 24px;display:flex;flex-direction:column;gap:9px}.eyebrow{font-size:10px;font-weight:600;letter-spacing:1.7px;color:#a4afa5;line-height:1.6}.sidebar>.eyebrow{padding:0 12px;margin-bottom:14px}.nav-item{display:flex;gap:12px;align-items:center;text-align:left;background:none;padding:12px;border-radius:6px;font-size:13px;color:#bbc2bb}.nav-item.selected{background:#303b32;color:#d9eddf}.keyhint{font-size:9px;color:#b4c5b7;margin-left:auto}.sidebar-section{margin-top:33px;padding:0 12px}.file-item{display:flex;gap:10px;background:none;color:#c3c9c1;font-size:12px;padding:18px 0}.session-card{margin-top:auto;padding:60px 12px 0;font-size:11px;color:var(--muted);line-height:1.7}.session-card strong{font-weight:500;color:#d3d7cb;display:block}.session-symbol{font-size:22px;color:#a7b4a4;display:block;margin-bottom:8px}.session-card p{margin-top:7px}.workspace{padding:42px 40px 22px;max-width:1700px;width:100%;margin:0 auto;min-width:0}.workspace-heading{display:flex;align-items:end;justify-content:space-between;margin-bottom:30px;gap:20px}h1{font-weight:400;letter-spacing:-1.3px;font-size:34px;margin:9px 0 0}h2{font-size:14px;font-weight:500;margin:0}h3{font-size:23px;font-weight:400;letter-spacing:-.5px;margin:20px 0 10px}.workspace-label{font-size:10px;color:var(--muted);font-family:ui-monospace,monospace;padding-bottom:5px;white-space:nowrap}.panels{display:grid;grid-template-columns:minmax(330px,1.3fr) minmax(300px,1fr);gap:19px;align-items:start}.panel{border:1px solid var(--border);border-radius:10px;background:var(--panel);overflow:hidden}.panel-header{display:flex;justify-content:space-between;align-items:center;border-bottom:1px solid var(--border);padding:18px 20px}.accent{color:var(--accent);margin-right:8px}.small-status{font-size:10px;color:var(--muted)}.activity{height:330px;overflow:auto;padding:23px}.welcome{text-align:center;padding-top:29px}.orb{margin:auto;border-radius:50%;width:55px;height:55px;display:grid;place-items:center;font-size:35px;color:var(--accent);background:radial-gradient(circle,#334c3e,transparent 75%)}.welcome p{font-size:12px;line-height:1.9;color:var(--muted)}.suggestion{background:#29342d;border:1px solid #425148;border-radius:6px;font-size:11px;padding:11px 14px;margin-top:14px;color:#cbdcd0}.suggestion span{margin-left:13px}.composer{margin:0 16px 16px;border:1px solid #4a574c;background:#272e28;border-radius:8px;padding:13px}.composer textarea{width:100%;resize:vertical;background:transparent;color:#f0f1e7;border:0;font-size:13px;line-height:1.6;min-height:66px}.composer textarea::placeholder{color:#a7b1a4}.composer-footer{display:flex;justify-content:space-between;align-items:center;gap:8px;margin-top:10px}.composer-footer>span{font-size:10px;color:var(--muted)}.primary,.secondary{border-radius:5px;padding:8px 13px;font-size:12px}.primary{background:#b7dec5;color:#1d3024;font-weight:600}.primary span{margin-left:12px}.secondary{background:#354238;color:#e1eadf;border:1px solid #536353}.tool-tabs{display:flex;border-bottom:1px solid var(--border);height:55px;padding:0 15px;gap:19px}.tool-tabs button{background:none;color:var(--muted);font-size:11px;padding:0 5px;border-bottom:2px solid transparent}.tool-tabs button[aria-selected=true]{color:#e0e8dc;border-bottom-color:#a3cdb0}.terminal-meta{display:flex;align-items:center;justify-content:space-between;color:#a4afa5;font-size:10px;padding:16px 17px}.text-button{background:none;color:#b6d5c0;font-size:10px;padding:4px}pre{white-space:pre-wrap;overflow-wrap:anywhere;font:12px/1.8 ui-monospace,SFMono-Regular,Consolas,monospace;color:#c5d4c5;margin:0;padding:12px 19px}#terminal-output{height:300px;overflow:auto}#terminal-form{display:flex;gap:10px;padding:13px 19px;border-top:1px solid #303830}.terminal-prompt{color:#9eccaa;font-family:monospace}#command{width:100%;min-width:0;border:0;background:transparent;color:#dee8d8;font:12px ui-monospace,monospace}.terminal-note{font-size:10px;line-height:1.6;color:var(--muted);margin:0;padding:0 19px 15px}#file-content{min-height:390px}.workspace-footer{display:flex;justify-content:space-between;align-items:start;gap:20px;margin-top:24px;color:#a3aba2;font-size:10px}.dev-controls{max-width:310px;text-align:right}.dev-controls summary{cursor:pointer}.dev-controls label{display:block;margin:14px 0 7px}.dev-controls select{max-width:100%;background:#2c362e;color:#e3e8db;border:1px solid #536353;border-radius:4px;padding:8px;font-size:11px}.dev-controls p{line-height:1.6}#preview-agent{display:block;margin:10px 0 6px}.funding{display:flex;justify-content:space-between;gap:30px;margin-bottom:20px;padding:25px}.funding h2{font-size:23px;margin:8px 0}.funding p{font-size:12px;color:#b8c2b7;line-height:1.7}.payment-steps{display:flex;flex-wrap:wrap;gap:8px;padding:0;list-style:none;margin:19px 0}.payment-steps li{font-size:10px;padding:7px;border:1px solid #465348;border-radius:4px;color:#a4afa5}.payment-steps li.done{background:#344e3e;color:#d5edda}.qr-card{min-width:170px;text-align:center}.qr-placeholder{width:92px;height:92px;margin:0 auto 10px;display:grid;place-content:center;border:8px double #b6c8b8;background:#26332b;color:#dce9da;letter-spacing:3px;font:12px/1.7 monospace}.qr-card strong{display:block;font-size:11px}.qr-card code{display:block;font-size:9px;margin-top:8px;color:#bcc8b9}.qr-card p{font-size:9px}.message{margin:0 0 20px;font-size:12px;line-height:1.8;white-space:pre-wrap}.message strong{display:block;color:#abd7bb;font-size:10px;letter-spacing:1px;text-transform:uppercase;margin-bottom:5px}.message.user strong{color:#d8cfb8}.message.system{color:#c9b88f;font-size:11px}.message.preview{color:#b9ad91;font-size:11px}.message.preview strong{color:#cdbf9d}.message.tool{color:#b7c3b4;font-size:11px}.message.tool strong{color:#93b3a0}@media(min-width:1500px){.workspace{padding-top:60px}.activity{height:410px}#terminal-output{height:380px}}@media(max-width:1100px){.sidebar{padding-left:10px;padding-right:10px}main{grid-template-columns:165px 1fr}.keyhint{display:none}.workspace{padding:30px 22px}.panels{grid-template-columns:1fr}.activity{height:290px}#terminal-output{height:200px}.workspace-label{display:none}.tools-panel{margin-top:0}}@media(max-width:650px){.topbar{padding:0 17px;height:69px}.wordmark{font-size:17px}.wordmark-light{display:none}.top-actions{gap:10px}.status{font-size:10px}.credit{font-size:11px;padding:8px}.banner-detail{display:none}main{display:block}.sidebar{display:none}.workspace{padding:28px 15px}.workspace-heading{margin-bottom:22px}h1{font-size:29px}.eyebrow{font-size:8px}.panels{grid-template-columns:minmax(0,1fr)}.funding{flex-direction:column}.qr-card{display:none}.composer-footer>span{max-width:150px}.workspace-footer{font-size:9px}} + +.credit-group{display:flex;flex-direction:column;align-items:end;gap:5px}#credit-freshness{font-size:9px;color:var(--muted)}@media(max-width:650px){#credit-freshness{font-size:8px}.top-actions{gap:7px}.status{font-size:9px}} +@media(max-width:650px){.wordmark{white-space:nowrap;font-size:15px}.mark{font-size:24px;margin-right:6px}.topbar{padding:0 12px;gap:10px}.credit{white-space:nowrap;font-size:10px}.status{max-width:95px}.top-actions{min-width:0}} + +/* Retained terminal sessions: one tab per PTY; Close acts on the selected one. */ +.terminal-tabs{display:flex;gap:6px;flex-wrap:wrap;padding:12px 17px 0;border-bottom:1px solid var(--border)}.terminal-tabs:empty{display:none;padding:0;border-bottom:0}.terminal-session-tab{background:#242a25;border:1px solid #363c37;border-bottom:0;border-radius:6px 6px 0 0;color:#a4afa5;font-size:10px;padding:6px 10px}.terminal-session-tab[aria-selected=true]{color:#e0e8dc;background:#2d372f;border-color:#4e6154}.terminal-actions{display:flex;gap:8px} +/* Narrow workbench (sidebar visible): the left navigation switches between the + agent and terminal panes so composers stay within the available space; wider + screens keep the split view and below 651px the stacked layout is preserved. */ +@media(min-width:651px) and (max-width:1100px){body[data-pane=agent] .tools-panel{display:none}body[data-pane=tools] .agent-panel{display:none}.workspace{padding:26px 22px 16px}.workspace-heading{margin-bottom:16px}h1{font-size:25px;letter-spacing:-.8px}}