diff --git a/AGENTS.md b/AGENTS.md index aa980e2e8..5f7c4e8e8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -117,9 +117,9 @@ and deserialized through a registry. | Package | Contents | |---|---| -| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`, one site: a server image, `backend_image` in its env.toml, serving the pages and API, checked at its `health` path; a separate frontend image is deprecated, warned of at put and optional in a bundle, and a stored `backend_docker_image_artifact` reads as the server image), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]` (`env/envs/computer/utils/install.py`): files, apt `packages` with extra `sources`, a `python` requirements file for a venv, `downloads` by URL and sha256, `run` scripts, the services the base's init runs, and `ports`; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch on a builder of the put's sandbox provider, by default the env's (`utils/builder.py`; `--sandbox` on `env build`, `bundle register` and `run`): this machine's Docker for `local`, else a VM of the provider, which fetches the put's stored files by signed URL, takes the rest inline and pushes to the image store itself, so it builds only with stores a remote VM reaches; each arch goes to the first link that runs it natively, else one that emulates it (a VM registers QEMU; `--no-emulate` leaves it `skipped` instead, for a later put), and its pair records the provider that built it (`builder`); a run or register that reuses a computer env composes what it has left (`pending`), a run only the forms its sandbox's deploys boot (`booted_forms`: on each link, the first form it boots natively, by the deploy's own rule, `boot.first_native`; a compose of a disk adds its container), and a deploy composes first what it boots that no compose started (`ComputerEnv.compose_unstarted`); tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile, build context and lock so any env with the same inputs reuses it; every layer's packages, requirements and downloads are resolved at put in a container of the base for each arch (`utils/packages.py`, `utils/resolve_in_base.py`), stored in the object store by sha256 under `agentenv-packages/`, and listed in a lock stored by a digest of its inputs, which any env with the same inputs reuses, and builds install only the lock's files, every install step `RUN --network=none`; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 (`utils/probing.py`) then boots each native build on the put's sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`, which declares each OS with its own decorator (`@linux(...)`, or `@install(os_type, spec)` for any OS type) holding its install, its launch and, optionally, a skill and settings and build settings that add to the class's there, and install steps and ready checks for every OS or those their `os=` names; everything that reads its settings, launch or steps asks for the computer's OS type; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference, over its env.toml's `[settings]` and a computer env's `settings` beside it in `apps`, each by top-level field (`params.app_settings`; checked at put, a required one left to the deploy; the deploy record keeps the merged values as `app_settings`, secrets withheld); its env.toml's `build_settings`, values of its `BuildSettings` checked at put, are written to `.agentenv/build_settings.json` in its folder, so they're part of the image's key, its install steps read them, and a deploy can't change them). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. Its site route, `/website//`, never forwards a site's data plane or card (`/agentenv/…`, `/.well-known/agent-env.json`), which agentenv reaches through `/svc//`; a site with no frontend image is served by its server image there. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | +| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`, one site: a server image, `backend_image` in its env.toml, serving the pages and API, ready when its card's `website/v1` `ready` answers a `WebsiteReadiness` that says so, as `agentenv_protocol.website` (`website_readiness`, `env_readiness`) defines a website env's health for the deploy and `DeployedEnv.readiness()` alike, or, for a card without `ready`, when its `health` path answers 200, which is deprecated; the gateway container's `env/gateway/constants.py` mirrors the URI and op, a test keeping them equal; a separate frontend image is deprecated, warned of at put and optional in a bundle, and a stored `backend_docker_image_artifact` reads as the server image), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]` (`env/envs/computer/utils/install.py`): files, apt `packages` with extra `sources`, a `python` requirements file for a venv, `downloads` by URL and sha256, `run` scripts, the services the base's init runs, and `ports`; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch on a builder of the put's sandbox provider, by default the env's (`utils/builder.py`; `--sandbox` on `env build`, `bundle register` and `run`): this machine's Docker for `local`, else a VM of the provider, which fetches the put's stored files by signed URL, takes the rest inline and pushes to the image store itself, so it builds only with stores a remote VM reaches; each arch goes to the first link that runs it natively, else one that emulates it (a VM registers QEMU; `--no-emulate` leaves it `skipped` instead, for a later put), and its pair records the provider that built it (`builder`); a run or register that reuses a computer env composes what it has left (`pending`), a run only the forms its sandbox's deploys boot (`booted_forms`: on each link, the first form it boots natively, by the deploy's own rule, `boot.first_native`; a compose of a disk adds its container), and a deploy composes first what it boots that no compose started (`ComputerEnv.compose_unstarted`); tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile, build context and lock so any env with the same inputs reuses it; every layer's packages, requirements and downloads are resolved at put in a container of the base for each arch (`utils/packages.py`, `utils/resolve_in_base.py`), stored in the object store by sha256 under `agentenv-packages/`, and listed in a lock stored by a digest of its inputs, which any env with the same inputs reuses, and builds install only the lock's files, every install step `RUN --network=none`; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 (`utils/probing.py`) then boots each native build on the put's sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`, which declares each OS with its own decorator (`@linux(...)`, or `@install(os_type, spec)` for any OS type) holding its install, its launch and, optionally, a skill and settings and build settings that add to the class's there, and install steps and ready checks for every OS or those their `os=` names; everything that reads its settings, launch or steps asks for the computer's OS type; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference, over its env.toml's `[settings]` and a computer env's `settings` beside it in `apps`, each by top-level field (`params.app_settings`; checked at put, a required one left to the deploy; the deploy record keeps the merged values as `app_settings`, secrets withheld); its env.toml's `build_settings`, values of its `BuildSettings` checked at put, are written to `.agentenv/build_settings.json` in its folder, so they're part of the image's key, its install steps read them, and a deploy can't change them). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. Its site route, `/website//`, never forwards a site's data plane or card (`/agentenv/…`, `/.well-known/agent-env.json`), which agentenv reaches through `/svc//`; a site with no frontend image is served by its server image there. A deploy with `capture_traffic` (`deploy_env`'s, `env deploy --capture-traffic`; a website env, or a multi env with a website, on a VM, refused for any other env and in container mode) records every packet each website's server and frontend containers send and receive from just after each starts (what a server sends while it starts up can come before its capture attaches): beside each runs a capture service, tcpdump from the gateway's image in that container's network namespace (`network_mode: "service:"`), the only service holding `NET_RAW` and `NET_ADMIN`, rotating its classic pcap files (10 of 20 MB each, the oldest overwritten) onto a volume no website container mounts and the gateway mounts read-only and serves as `urn:agentenv:traffic/v1` (`list`, `GET /traffic/captures`, oldest first; `get`, `GET /traffic/capture?name=`), which `agentenv_protocol.traffic` defines (`TrafficCapture`, `list_traffic_captures`, `fetch_traffic_capture`), the gateway's `constants.py` mirroring its names, a test keeping them equal. Those routes take the per-deploy control token the record keeps (`DeployedEnv.control_token`, the gateway's `AGENTENV_CONTROL_TOKEN`), which the gateway takes off every request before it proxies one, so no site reads it; a capture that isn't running once the gateway is up refuses the deploy. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | | `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Every artifact but an environment universe, whose `metadata` key holds its legacy metadata file ids, carries a `metadata` dict outside its versioned fields, changed by compare-and-swap (`merge_metadata`). Built-ins: `CliArtifact` (`cli`), `ComputerArtifact` (`computer`, a base computer: its architectures, OS, session user, settings and container form, whose image `put` reads from its registry into `metadata.validated_images`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`: its environment's files by path, stored as the one wrapped `file_artifact_ref` every environment artifact had when it has one file at its filename, else as `file_artifact_refs`; `get_file_artifacts()` reads either, and a load hands each file to the env's `data/add` with its path, which a pre-v1 load, taking one file, refuses), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or a published image's tag with its `image_digest` and its build context, which sandboxes pull by the digest and build from the context when they can't (the infra images, and a bundle agent built from a shipped agent's sources, `Plan.published`), or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/docker/context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. | -| `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, among them `website_state_verifier`, which reads each checked session of a deployed website env's sites, or a multi env's, at the site's `/go?sid=` through `/svc//` and checks one dot path of its `current_state`, `initial_state` or `state_diff` each, optionally reduced by `len`, `keys` or `bool`, with `eq`, `ne`, `ge`, `gt`, `le`, `lt`, `in` or `contains`, its sites checked before a run against the `env_version` it pins, `start_computer_recording` and `end_computer_recording`, and more). On an env whose card serves computer-exec/v1 (a computer), `run_code` with `env_id` and `run_container_unit_tests_verifier` with `env_id` run inside it as its session user through the driver: files staged with `data/add` and read with `data/get`, commands run with `run`; `env_outcome_verifier` hands a `verify` that takes an `env_url` keyword the env's address. A computer's framework-only routes (everything but MCP and the card) take the per-deploy control token its provider boots it with (`DeployedEnv.control_token`, `AGENTENV_CONTROL_TOKEN`); framework calls send `deployed.control_headers`, `Task.run` runs every step under `protocol_v1.authorized` with each deployed env's (a `verify` included), and a `run_code` script inside the computer gets `input["env"]["headers"]`. `teardown_run` and the `teardown_sandboxes` step share one core (`task/teardown.py` `tear_down`): the recordings the run left open end with fresh grants, each env still up closes (`Env.close`; a computer env ends any recording still open), what stayed open is registered from its namespace, then every sandbox is terminated by id. `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | +| `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, among them `start_computer_recording` and `end_computer_recording`, `collect_traffic_capture`, which stores each traffic/v1 capture of an env deployed with `capture_traffic` as a pcap `FileArtifact` in one `FileArtifactUniverse` recorded under `context.metadata["traffic_captures"][env_id]`, and more). On an env whose card serves computer-exec/v1 (a computer), `run_code` with `env_id` and `run_container_unit_tests_verifier` with `env_id` run inside it as its session user through the driver: files staged with `data/add` and read with `data/get`, commands run with `run`; `env_outcome_verifier` hands a `verify` that takes an `env_url` keyword the env's address. A computer's framework-only routes (everything but MCP and the card) take the per-deploy control token its provider boots it with (`DeployedEnv.control_token`, `AGENTENV_CONTROL_TOKEN`); framework calls send `deployed.control_headers`, `Task.run` runs every step under `protocol_v1.authorized` with each deployed env's (a `verify` included), and a `run_code` script inside the computer gets `input["env"]["headers"]`. `teardown_run` and the `teardown_sandboxes` step share one core (`task/teardown.py` `tear_down`): the recordings the run left open end with fresh grants, each capturing env still up whose traffic the run didn't collect is collected, a retry's rolled-back ones too (kept as `capturing_envs` in its `retry_resets` marker, each under `-rolled-back-`), and what it records saved to the stored instance's `context.metadata`, each env still up closes (`Env.close`; a computer env ends any recording still open), what stayed open is registered from its namespace, then every sandbox is terminated by id. `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the local namespaces, `@local` and `@hf` (`is_local_id`), each with a documents file of its own: while the CLI or a `build_app` server runs, a local id and what a local task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | | `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra), `smol_vm` (the `smol` extra), `tensorlake` (the `tensorlake` extra), `tenki` (the `tenki` extra), `vercel` (the `vercel` extra), `boxd` (the `boxd` extra), `coreweave` (the `coreweave` extra), `daytona` (the `daytona` extra), `boat` (the `boat` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain, whose deploys skip a provider that can't run them. A sandbox provider declares what it runs (`ON_THIS_MACHINE`, and for VMs on this machine the address their requests to it arrive at, `REACHES_HOST_AT`; `CONTAINER_ACCEPTS`, `PRIVATE_NETWORK`; `SUPPORTED_SANDBOX_MODES`, derived from whether it implements `create_sandbox` and `create_vm`; and `SANDBOX_ACCEPTS`, the images an agent's sandbox runs, which defaults to those a VM loads when it implements `create_vm` and to `CONTAINER_ACCEPTS` otherwise, so only a provider that implements `create_vm` yet gives an agent a container, like the local one, declares it), each sandbox carries the `platform` of the images it runs (linux/amd64 unless its provider sets another when it creates it; None, any, unchecked, for a local sandbox), and code outside `providers/sandbox_providers/` reads those, never which provider it has. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store), `EnvironmentGatewayProvider` (`gateway`, an env's default), which renders a docker-compose for the gateway and its MCP servers inside the sandbox, and `EnvironmentServerProvider` (`server`), one MCP server in its own container with no gateway in front and no state store; `providers/env_state/` holds env-state providers (`local_postgres` built in). | diff --git a/packages/agentenv-protocol/README.md b/packages/agentenv-protocol/README.md index b6cc8c649..847455739 100644 --- a/packages/agentenv-protocol/README.md +++ b/packages/agentenv-protocol/README.md @@ -198,6 +198,16 @@ with any other transport are kept beside the SDK's entry. Card URLs are paths, r address the card was fetched from. On the client side, `client.mcp_path(card)` returns the declared path, or `/mcp` for a card without one. +A composed card, such as a gateway's, nests each backing environment's card under +`children_environments`, its `url` and endpoints rooted at the route the composer serves it at. +`client.children(card)` returns each as a `ChildEnv` (its `name`, `url` and `card`, with its +`extensions`, `operations` and `serves(method)`, and `base_url(address)`, its data plane resolved +against the composed card's address). Two children may share a name, as a website and an MCP +server can, so `client.child_at(card, url)` finds one by its url or route; `client.child_named(card, +name)` raises `AmbiguousChildError` for a shared name, and `client.find_child` returns the first. +`client.serves(card, method)` says whether a card serves a data-plane method by its declared +`operations`, None for a card that declares none. + Extensions are invoked from what the card advertises. `client.find_extension_method(card, uri, method)` returns one advertised method, whose `endpoint` is the method's own or else the extension's, and `client.invoke_extension(base_url, card, uri, params, method=...)` calls it with @@ -420,6 +430,33 @@ async with ToolSession(mcp_url=mcp_url, role="alice@example.com") as session: - **Tools:** `list_tools()` returns the tools the role may use as `EnvironmentTool`, the type a card's `capabilities.tools` holds. Each `inputSchema` is the JSON Schema a model API takes as the tool's parameters. +## Website readiness + +`agentenv_protocol.website` is the one definition of whether a website env is healthy. A site's card advertises +`urn:agentenv:website/v1`'s `ready` (`WEBSITE_READY_OP`), a GET answering a `WebsiteReadiness`: `ready`, a boolean, +and `problems`, the names of what keeps it from being ready (a `WebsiteEnvironment`'s are its `@ready_check` methods +not yet true). The model is strict about those two (`"yes"` is no `ready`) and ignores fields it doesn't know, so a +reader keeps reading a newer site. Everything that judges a site reads it through these helpers: agent-env's deploy, +`DeployedEnv.readiness()`, and the conformance checker's `health`, which fails an answer that isn't a +`WebsiteReadiness`. + +```python +from agentenv_protocol.website import env_readiness, website_readiness + +site = await website_readiness(base_url, card) # WebsiteReadiness, or None for a card without ready +env = await env_readiness(gateway_url, env_card) # WebsiteReadiness of every site of the env +print(env.ready, env.problems) # False ['shop: seeded'] +``` + +- **One site:** `website_readiness(base_url, card, *, timeout=READY_TIMEOUT_S, headers=None)` asks the site as its card + advertises `ready`. It raises httpx's error when the site can't be reached, `ExtensionCallError` for a non-2xx + answer and `ValueError` (pydantic's `ValidationError`) for an answer that isn't a `WebsiteReadiness`. +- **An env:** `env_readiness(base_url, card, ...)` asks, at once, the card itself when it advertises `ready` (a single + site's card), and every child under a composed card's `children_environments` that does, its endpoints rooted at + `base_url` as a gateway's are. A child's problems are prefixed with its name (`shop: seeded`); one that can't be + asked, or answers no `WebsiteReadiness`, is a problem itself. A card or child without `ready`, an MCP server's or an + older site's, isn't counted, so an env with no site serving it is ready with no problems. + ## Computer drivers A computer driver is what runs inside a computer an agent uses through screenshots and input. Write diff --git a/packages/agentenv-protocol/docs/website-spec.md b/packages/agentenv-protocol/docs/website-spec.md new file mode 100644 index 000000000..664b90342 --- /dev/null +++ b/packages/agentenv-protocol/docs/website-spec.md @@ -0,0 +1,359 @@ +# The website spec + +A website env used to be described in three places: its `env.toml` (`type = "website"`, `health`, `backend_image`), +the decorators in its code (`@website(domains, static, health, data)` and `@api` routes with names and roles), and the +card it serves (the `urn:agentenv:website/v1` extension: domains, health, routes, session cookie). The website spec +is one declaration of a site. Each way of serving a site either produces a spec or is checked against one, and specs +compose, so a site can be assembled from parts. + +The code is `agentenv_protocol.website.spec`. The JSON Schema of a spec file is +`python -m agentenv_protocol.website.spec schema`. + +## Shape + +A spec is a TOML file. Its top-level keys are fixed and every section is optional: + +| Section | What it says | +| --- | --- | +| `extends` | The path of a spec this one overlays (see [Overlays](#overlays)). | +| `[site]` | `name`, the card's name; `domains`, the host names the site answers for, with `*.` as the leftmost label matching any subdomain; `health`, the path that answers 200 once the site is usable (`/api/health` by default), read only for a server whose card has no `website/v1` `ready`, which is deprecated: a deploy and the conformance check read readiness from the card. | +| `[source]` | What serves the site, exactly one of `image`, `sdk`, `static`, `live` or `replay` (see [Sources](#sources)). | +| `[data]` | `collections` (each with `key`, `owner` and an optional JSON `schema`), or one `schema` of the whole state, or `models = "mcp:"` to bind the site's data to an MCP server's schema; and `seed`, the files it starts with, in order. | +| `[users]` | `personas` (ids, or `{id, name, role}` tables); `acting_user`, who a browser with no session is; `sign_in`, which is `cookie` (the site's signed session cookie, by default), `none` or `live`; and, for a cookie sign-in by a server not built on the SDK, its `session_cookie` and `session_path`. | +| `[roles]` | A role name mapped to the routes it may call. These roles are added to each route's own `roles`. | +| `[routes]` | `ops`, the named operations, each `name = "METHOD /path"` or a table `{method, path, roles, served_by}`; and `openapi`, an OpenAPI document whose operations, named by `operationId` (each name once) with roles from `x-agentenv-roles`, are routes too. Triggers and records call these names `op`. | +| `[clock]` | How the site follows the universe's time: `follow = "header"` (`X-Universe-Now` on every answer), `"date_patch"` (its pages' `Date` is shifted too), or `"none"` (wall time). | +| `[[uses]]` | Components: another spec mounted under a path prefix (see [Composition](#composition)). | + +Paths in a spec (Dockerfiles, folders, seeds, schemas, captures, the OpenAPI document and components) are relative to +the spec file. + +The Notes example's spec, `examples/website/notes/website.toml`: + +```toml +[site] +name = "notes" +domains = ["notes.test"] +health = "/healthz" + +[source] +sdk = "site:Notes" +assets = "web/dist" + +[data] +seed = ["data/seed.json"] + +[data.collections.users] + +[data.collections.notes] +owner = "owner" + +[users] +personas = [{ id = "ann", name = "Ann Lee" }, { id = "ben", name = "Ben Ortiz" }] +acting_user = "ann" +sign_in = "cookie" + +[roles] +admin = ["all_notes"] + +[routes.ops] +list_users = "GET /api/users" +sign_in = "POST /api/sign-in" +list_notes = "GET /api/notes" +create_note = "POST /api/notes" +get_note = "GET /api/notes/{note_id}" +edit_note = "PATCH /api/notes/{note_id}" +toggle_note = "POST /api/notes/{note_id}/toggle" +delete_note = "DELETE /api/notes/{note_id}" +all_notes = "GET /api/admin/notes" + +[clock] +follow = "header" +``` + +## Validation + +`load(path)` reads a spec, resolves `extends` and `uses`, and validates the result. If anything is wrong, it raises +`SpecError`, whose `problems` lists every problem as `: : `. For example, +`.../shop.toml: routes.ops: b and a are both GET /x`. The checks within a section are: unknown keys, exactly one +source, lower-case host names, health and route paths outside the data plane's `/agentenv` and the card's +`/.well-known/agent-env.json`, credentials that are `secret:`/`env:` references rather than values, and an +`acting_user` that is one of the `personas`. + +The checks across sections are: + +- No two routes share a verb and path, and no GET route is served at the health path. +- `[roles]` names only routes that exist, including the namespaced routes of components. +- `sign_in = "live"` needs a `live` or `replay` source. +- A `live` or `replay` site is read as `diff`, and only such a site is. +- An `sdk` site follows the clock, so `follow = "none"` is refused for it. +- A `static` site serves no routes. +- `served_by` is allowed only on a route of a live or replayed site. +- A collection with an `owner` needs a site that signs users in. + +`python -m agentenv_protocol.website.spec validate PATH` runs these checks. `show PATH` prints the resolved spec as +JSON, and `show --card PATH` prints the card params it advertises. + +## Sources + +Every source maps onto the same spec. Each row below says how a source is described, what it produces, and what it is +checked against. + +| Source | `[source]` | Where the spec comes from | Deployable today | +| --- | --- | --- | --- | +| SDK class | `sdk = "module:Class"`, `assets = ""` | `@website(spec="website.toml")` declares the class from the spec, and the class is refused if it doesn't serve what the spec declares. `from_environment(cls)` derives a spec from the decorators of a class declared the old way. | Yes, as a website env's server image | +| Any-stack image | `image = ""` or `image = { dockerfile, context }` | Written by hand. `conformance --spec` checks the running server against it. | Yes, through `backend_image` in env.toml | +| Static | `static = ""` | Written by hand. | Modeled only: no generic static server image ships yet | +| Live real site | `live = { allow_hosts, credentials }` | Written by hand; `from_card` reads one back from a proxy that serves a card. | Modeled only; a website env refuses it | +| Replayed capture | `replay = { capture, allow_hosts }` | Usually an overlay of the live spec. | Modeled only; a website env refuses it | + +For every source, `to_card_extension()` returns the `urn:agentenv:website/v1` params a server of the spec advertises: +`domains`, `health` and `routes` (each `{method, path, name, roles}`); `session_cookie` and `session_path` for a cookie +sign-in; and `clock` when it is declared. A WebsiteEnvironment builds its card from these params, so +the card and the spec cannot drift apart. `from_card(card)` reads a running site's card back into a spec. It recovers +the site, routes, sign-in and clock. A card does not carry a site's source, data, personas or the +`[roles]` table; roles are folded into each route. + +### The SDK + +```python +@website(spec="website.toml") +class Notes(WebsiteEnvironment): + ... +``` + +`@website(spec=...)` takes a path relative to the class's file, or a loaded `WebsiteSpec`. The spec supplies the +domains, health path, static folder (`assets`), seed files and clock patch. If any of those are also passed to +`@website`, it is refused. When the class is declared, it is checked against the spec: + +- Its `@api` routes must match the spec's routes by name, verb, path and roles, whenever the spec declares routes. +- `source.sdk` must name the class. +- The sign-in must be the SDK's cookie. +- `acting_user` must be the seed's `current_user`. + +If the class has no `@environment_card` name, the card is named by `site.name`. A card name that differs from +`site.name` is refused. Classes declared with `@website(domains=..., ...)` work as before; their card goes through +`from_environment(cls).to_card_extension()`. + +### env.toml + +A website env's `env.toml` can point to a spec with `spec = "website.toml"`, or carry one inline as a `[spec]` +table: + +```toml +type = "website" +spec = "website.toml" +backend_image = { dockerfile = "Dockerfile" } +``` + +The env takes its `environment_name` from `site.name` and its health path from `site.health` when the env.toml leaves +them out. If the env.toml sets one to a different value, the env is refused. The env stores the spec's domains +(`WebsiteEnv.domains`). A spec with a `live` or `replay` source is refused, because a server image can't serve it. +`backend_image` still says how the image is built. + +### Conformance + +`python -m agentenv_protocol.website.conformance --spec website.toml` runs the usual checks plus two more: + +- `spec` checks that the card's name, domains, health path, routes, session cookie and session path are the spec's, + and its `clock` too when the spec declares it. +- `spec_routes` checks that every declared route is served. An `OPTIONS` to the route's path, with each path parameter + filled in by a value the converter of the card's route of that name takes (a UUID for `uuid`, else `0`), must find + something there; a GET route without path parameters must answer neither 404 nor 405; and each + route must be an operation of the server's OpenAPI document when its card offers one. When the card offers no + document, a route only the `OPTIONS` can check (a non-GET route, or a GET route with path parameters) counts as + served only when that `OPTIONS` answers below 500 with an `Allow` listing its method. Any other answer, such as a + catch-all `OPTIONS` handler's 200 without `Allow`, a 500, or an `Allow` that doesn't list the method, leaves the + route unverified: the check is then a `warning` (a failure with `--strict`) rather than a pass, and fails only + when a route is shown missing. Only GET routes are called directly, so the check changes no state. + +The `health` check reads readiness from the card's `website/v1` `ready`. For a card without it, it reads the health path, which is deprecated: with `--spec`, the spec's rather than the card's. + +## Overlays + +`extends = ""` merges this spec over the one it names. The merge works like this: + +- Tables merge key by key, recursively. +- Arrays and scalars replace. For example, `domains = [...]` or `personas = [...]` in the overlay replace the base's. +- `[source]` replaces as a whole, so an overlay can turn a live site into a replay without leftover keys. +- `name = false` removes an inherited entry of `[routes.ops]`, `[roles]` or `[data.collections]`. A route removed + this way is removed from the operations of the `openapi` document too. +- A route written as `"METHOD /path"` is expanded to a table before merging, so an overlay can change one key of it, + such as `post_message = { served_by = "..." }`. An operation of the `openapi` document is one too: a table naming it, + such as `add_item = { roles = ["staff"] }`, is written over the document's operation. +- `[[uses]]` entries are added. An entry with the same `as` (or, without one, the same `spec`) replaces the inherited + entry. +- The base's relative paths are rewritten relative to the overlay's folder, so a base in another folder still finds + its files. + +`extends` chains are followed, and a cycle is refused. In code, `spec.overlay({...})` applies the same merge to a +loaded spec. The Notes test uses it to swap in a stand-in static folder. + +Overlays are how one site gets variants: + +- A frozen capture with one synthetic route: `examples/website/specs/slack.toml` extends `slack-live.toml`, replaces + the live source with a replay of a capture, and swaps `post_message` to an SDK class. +- The same site with another persona set: `examples/website/specs/intranet-ben.toml` signs in as `ben`, with personas + `ben` and `cy`, and drops the `editor` role. + +```toml +# slack-live.toml +[site] +name = "slack" +domains = ["slack.com", "*.slack.com"] +health = "/api/api.test" + +[source.live] +allow_hosts = ["*.slack-edge.com", "*.slack-imgs.com"] +credentials = { token = "secret:SLACK_USER_TOKEN" } + +[users] +sign_in = "live" + +[routes.ops] +list_conversations = "GET /api/conversations.list" +history = "GET /api/conversations.history" +post_message = "POST /api/chat.postMessage" +add_reaction = "POST /api/reactions.add" + +[clock] +follow = "none" +``` + +```toml +# slack.toml: the live site frozen as a replay, one route answered by a synthetic implementation +extends = "slack-live.toml" + +[source.replay] +capture = "captures/slack-workspace.har" +allow_hosts = ["*.slack-edge.com"] + +[routes.ops] +post_message = { served_by = "slack_synthetic:SlackChat" } + +[clock] +follow = "date_patch" +``` + +## Composition + +A `[[uses]]` entry mounts another spec as a component: + +```toml +[[uses]] +spec = "components/sso.toml" # relative to this spec +mount = "/sso" # the path prefix its routes are served under +as = "sso" # its namespace; the component's site.name by default +data = { accounts = "users" } # a collection of the component's, mapped onto one of this site's +``` + +When a spec is resolved, each component is resolved first, including its own `extends` and `uses`. It is then merged +into the site: + +- **Routes.** Each of the component's routes is served at `` and named `.`. The roles its own + `[roles]` grants come with it. The site's `[roles]` can name the composed routes, for example + `editor = ["sso.login"]`. +- **Data.** A collection named in `data` merges into the site's collection of that name. They must agree on `key`, + `owner` and `schema` where both set them. Any other collection is added as `.`. A schema file a + component names stays the component's file, its path made relative to the site's spec. A component's seed files + are not merged: the site seeds the shared collections. +- **Users.** The component's personas are added to the site's, merging the fields of a persona they share; fields + that contradict are a conflict. A component must sign in the way the site does, or not at all (`none`). +- **The rest.** The component's `site` (domains, health), `source` and `clock` are not merged. A mounted + component lives under the site's own domains and is served by the site's source. + +These are conflicts, each reported as `uses[]: ...`: + +- A composed route name that the site already declares. +- A composed verb and path that another route already serves. +- A route of the site's own under the component's mount prefix (unless the mount is `/`). +- A `data` mapping of a collection the component doesn't declare. +- A collection or persona that disagrees with the site's. +- A sign-in mismatch. +- A component with no name. +- A cycle of `uses`. + +A resolved spec has no `extends`, `uses` or `routes.openapi` left: the document's operations are in `routes.ops`. +`spec.components` lists what was mounted, each with its name, mount and own resolved spec. `to_dict()` of a resolved +spec reads back as the same spec. + +`examples/website/specs/intranet.toml` uses a fake SSO component, `components/sso.toml`, mounted at `/sso`. The +component's `accounts` collection is mapped onto the intranet's `users`, and its `grants` collection becomes +`sso.grants`: + +```toml +[site] +name = "intranet" +domains = ["intranet.test", "*.intranet.test"] + +[source.image] +dockerfile = "Dockerfile.intranet" + +[data.collections.users] + +[data.collections.pages] +owner = "author" + +[users] +personas = ["ann", "ben"] +acting_user = "ann" + +[roles] +editor = ["publish_page"] + +[routes.ops] +list_pages = "GET /api/pages" +publish_page = "POST /api/pages" + +[[uses]] +spec = "components/sso.toml" +mount = "/sso" +data = { accounts = "users" } +``` + +Resolved, its routes are `list_pages`, `publish_page`, `sso.authorize` (`GET /sso/authorize`), `sso.login`, +`sso.token` and `sso.userinfo`, and its collections are `users`, `pages` and `sso.grants`. + +## The name table + +A multi env routes every host name its sites answer for through one table. `name_table(specs)` builds it: + +- **Kinds.** A site's `domains` are `synthetic` when the env serves the site (an image, sdk, static or replay source) + and `live` when the site is the real one. A live or replay source's `allow_hosts` are `allowlist`: hosts let through + to the internet, not served. +- **Precedence.** When two kinds claim one pattern, the higher kind wins: synthetic, then live, then allowlist. The + loser is kept in `shadowed` beside the entry it lost to. This is how a synthetic overlay takes over a name a live + site or an allowlist also claims. +- **Resolution.** `table.resolve(host)` looks first at the highest kind of the entries matching the host, then at the + most specific pattern of that kind: an exact name, then the longest wildcard. So a synthetic `*.slack.com` beats a + live `status.slack.com`. A synthetic `inbox.example.test` beats a synthetic `*.example.test` of another site. +- **Wildcards.** `*.example.test` matches every subdomain at any depth, but not `example.test` itself. + +These are conflicts, each a `SpecError`: + +- Two sites claim the same pattern at the same kind, synthetic or live, whichever kind wins the pattern. A host that + two different patterns match (`*.example.test` and `inbox.example.test`) is not a conflict, because specificity + decides it. +- A site has no `site.name`. +- Two sites share a name. + +Two sites allowlisting the same host is not a conflict, because an allowlist only lets traffic through. + +## What is wired and what is only modeled + +These parts are wired: + +- The spec model, loader, validation, `extends`, `uses`, the card params, `from_card`, `from_environment`, the name + table, the JSON Schema and the command line. +- `@website(spec=...)` and the SDK card generated through the spec. +- `spec` in a website env's env.toml (name, health and domains). +- `conformance --spec`. + +These parts are only modeled: they are declared and validated, but nothing serves them yet. + +- The `static`, `live` and `replay` sources, and route `served_by`. +- `[data] models` and collection schemas. +- Mounting a component's implementation. An SDK class that uses a component must serve its routes itself. +- The name table's use by the gateway. +- Component references other than file paths. +- The env's stored domains: deploy doesn't yet check them against the card. diff --git a/packages/agentenv-protocol/examples/website/notes/.gitignore b/packages/agentenv-protocol/examples/website/notes/.gitignore new file mode 100644 index 000000000..545a6e274 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/.gitignore @@ -0,0 +1,2 @@ +web/node_modules/ +web/dist/ diff --git a/packages/agentenv-protocol/examples/website/notes/Dockerfile b/packages/agentenv-protocol/examples/website/notes/Dockerfile new file mode 100644 index 000000000..f8896a549 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/Dockerfile @@ -0,0 +1,26 @@ +ARG NOTES=packages/agentenv-protocol/examples/website/notes + +FROM mirror.gcr.io/library/node:22-slim@sha256:4b150b48b4116e2a22f2c39cc27025fbf4f153e2078fd3b14d159fdb5c7279da AS web +ARG NOTES +WORKDIR /web +COPY ${NOTES}/web/package.json ${NOTES}/web/package-lock.json ./ +RUN npm ci --no-audit --no-fund +COPY ${NOTES}/web/ ./ +RUN npm run build + +FROM mirror.gcr.io/library/python:3.12-slim@sha256:423ed6ab25b1921a477529254bfeeabf5855151dc2c3141699a1bfc852199fbf +ARG NOTES +WORKDIR /app +RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/* +COPY packages/agentenv-protocol /tmp/agentenv-protocol +RUN pip install --no-cache-dir /tmp/agentenv-protocol "uvicorn>=0.30" && rm -rf /tmp/agentenv-protocol +COPY ${NOTES}/site.py ${NOTES}/website.toml /app/ +COPY ${NOTES}/data /app/data +COPY --from=web /web/dist /app/web/dist +ENV PORT=8000 +EXPOSE 8000 +RUN useradd --create-home --uid 10001 appuser +USER appuser +HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \ + CMD python -c "import os, urllib.request; urllib.request.urlopen('http://127.0.0.1:%s/healthz' % os.environ.get('PORT', '8000'), timeout=2)" +CMD ["python", "site.py"] diff --git a/packages/agentenv-protocol/examples/website/notes/Dockerfile.dockerignore b/packages/agentenv-protocol/examples/website/notes/Dockerfile.dockerignore new file mode 100644 index 000000000..10ae9270c --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/Dockerfile.dockerignore @@ -0,0 +1,6 @@ +**/node_modules +**/dist +**/__pycache__ +**/.pytest_cache +**/.venv +.git diff --git a/packages/agentenv-protocol/examples/website/notes/README.md b/packages/agentenv-protocol/examples/website/notes/README.md new file mode 100644 index 000000000..0d55da1c4 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/README.md @@ -0,0 +1,109 @@ +# Notes: a reference website/v1 site + +Notes is a small notes app built on `agentenv_protocol.website.WebsiteEnvironment`. It is the reference site for the +`urn:agentenv:website/v1` server contract. Each user keeps their own notes and can create, edit, mark done and delete +them. The page shows the universe's time. + +| File | What it is | +| --- | --- | +| `website.toml` | The site's spec: its name, domains, health path, source, seed, collections, personas, roles, routes, state read and clock (see `docs/website-spec.md`). | +| `site.py` | The server: the `Notes` WebsiteEnvironment, declared from `website.toml`, its `@api` routes, and `serve()` on `PORT` (8000). | +| `data/seed.json` | The seed data: two users (`ann`, `ben`), their notes, and `current_user`, the acting user. | +| `web/` | A minimal Vite app (`index.html`, `src/main.js`, `src/style.css`, and `src/site-client.js`, a copy of the SDK's browser client). Its assets (`base: "./"`) and API calls are relative to the page, so it works at `/` and under the gateway's `/website/notes/` alike; opened at a path without its final slash, such as `/website/notes`, the page first reloads itself with it. | +| `Dockerfile` | The server image: Node builds `web/`, then Python installs this repo's `packages/agentenv-protocol` with uvicorn. It installs `curl`, which the gateway's compose health check runs. | +| `env.toml` | The website env as a bundle entry, pointing at `website.toml`. | + +## What it exercises + +- **Site declaration.** `@website(spec="website.toml")`. The spec gives the name, domains, static folder, health path + and seed, and the class is refused when its `@api` routes aren't the spec's. The card's website/v1 params are the + spec's `to_card_extension()`: the domains, the health path, every route by name, the session, the state read and + the clock. +- **Routes.** `list_users`, `sign_in`, `list_notes` (with an optional `?done=`), `create_note` (201), `get_note`, + `edit_note`, `toggle_note` and `delete_note`. Errors are raised as `NotFound`, `BadRequest` and `Unauthorized`. +- **Roles.** `all_notes` (`GET /api/admin/notes`) is declared with `roles=["admin"]`, and the spec's `[roles]` grants + it to `admin`. It answers only a caller that + sends `AgentEnv-Role: admin` with `AgentEnv-Role-Key` equal to the container's `WEBSITE_ROLE_KEY`; any other caller + gets 403. A deploy with `keyed_roles` gives the site its own role key as `WEBSITE_ROLE_KEY` and its gateway sends it + on only beside a role it verified with the same key, so a page cannot claim `admin` through it. Any other deploy sets + no key, and its gateway passes on whatever role a browser sends; with no `WEBSITE_ROLE_KEY` set, every caller is + `default` and the route is closed. +- **Users.** `ApiContext.user` is the user in the signed session cookie, `agentenv_session_notes` (the site's own, so + it doesn't collide with another site's under the same gateway host; the card names it). With no cookie, it is the acting user (the + store's `current_user`), and `GET /api/session` signs that user in. `POST /api/sign-in` switches to another seeded + user. A user's notes are invisible to every other user, and another user's note answers 404. +- **State.** The site writes no data hooks, so it uses the default store. `data/add` loads the seed (as a data part, or + as a file part by bytes or `file://`), `data/get` exports it, and `data/reset` empties it. `data/get` is how + agentenv reads what a task left the site with. +- **Time.** Notes are stamped with `self.now()`, which follows the gateway's clock once `clock/v1`'s `sync_time` names + it. The frontend reads `X-Universe-Now` from every API response into `universeNow()` and shows it in the header. + It uses this helper rather than `clock_patch=True`, so the page works the same against a server written without the + SDK. The helper and the page's `api()`, `session()` and `signIn()` come from `src/site-client.js`, a copy of + `agentenv_protocol/website/client/site-client.js`, which a test keeps it the same as. +- **Readiness.** `@ready_check seeded` holds `/healthz` at 503 until the store has users, for example after a + `data/reset` and before the next `data/add`. + +## Run it + +From this folder: + +```bash +cd web && npm ci && npm run build && cd .. +PYTHONPATH=/packages/agentenv-protocol/src python site.py +``` + +Open http://localhost:8000. To work on the frontend, run `npm run dev` in `web/` while `site.py` runs; Vite proxies +`/api` to port 8000. + +Or run `agent-env env website dev` from this folder: it builds `web/` when it is out of date and rebuilds it as it +changes, serves `site.py` with reload, syncs it to a local stand-in for the gateway's clock, loads `data/seed.json` +through the data plane after every reload, and runs the conformance check once. + +## Test it + +```bash +python -m pytest packages/agentenv-protocol/tests/test_website_notes_example.py +``` + +The test imports `site.py`, runs the conformance checker, and checks sign-in, per-user notes, the admin-only route, +clock-synced timestamps, and the data plane's reset and seed load. It serves a stand-in page, so it needs no Vite +build. + +## Build and verify the image + +The build context is the repo root, because the image installs the protocol package from this repo's source: + +```bash +docker build -f packages/agentenv-protocol/examples/website/notes/Dockerfile -t notes-site . +docker run -d --name notes-site -p 18080:8000 notes-site +python -m agentenv_protocol.website.conformance http://localhost:18080 --clock-host host.docker.internal --strict +curl -s http://localhost:18080/api/notes +``` + +Use `--clock-host` so the container can reach the checker's clock, and add +`--spec packages/agentenv-protocol/examples/website/notes/website.toml` to check the server against its spec too. +`--strict` fails on a recommended check's warning too (the card, `X-Universe-Now`, the clock, forwarded headers, +cookies); without it only the contract's required checks, and the spec's when given, fail the run, and Notes passes +every check either way. The clock check leaves the server following a +clock that is gone once the check ends: later timestamps start from 2031 until the server is synced again, so restart +the container before using it. + +## Register it + +```bash +agent-env env website put --id notes-site \ + --server-dockerfile packages/agentenv-protocol/examples/website/notes/Dockerfile \ + --server-docker-context . +``` + +Run it from the repo root. `environment_name` comes from the card, which the spec names `notes`. The stored env +has a server image and no frontend image. Pass `--skip-validation` to register without deploying. + +To record what the site sends and receives, deploy it with `--capture-traffic` (or `"capture_traffic": true` on a +task's `deploy_env` step): tcpdump runs in the server container's network namespace and the gateway serves its pcap +files as `traffic/v1`, which a `collect_traffic_capture` step with the env's `env_id` stores as file artifacts. + +`env.toml` describes the same env for a bundle. Its keys are `type`, `spec` and `backend_image`; the env takes its +`environment_name`, health path and domains from the spec. A bundle builds `backend_image` with this +folder as the build context, but this Dockerfile needs the repo root. Until the protocol package is installed from a +published release instead, register the site with `put` as above. diff --git a/packages/agentenv-protocol/examples/website/notes/data/seed.json b/packages/agentenv-protocol/examples/website/notes/data/seed.json new file mode 100644 index 000000000..0a52ab7be --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/data/seed.json @@ -0,0 +1,15 @@ +{ + "current_user": "ann", + "users": [ + {"id": "ann", "name": "Ann Lee"}, + {"id": "ben", "name": "Ben Ortiz"} + ], + "notes": [ + {"id": 1, "owner": "ann", "title": "Buy milk", "body": "Oat, two cartons.", "done": false, + "created_at": "2026-01-05T09:00:00.000Z", "updated_at": "2026-01-05T09:00:00.000Z"}, + {"id": 2, "owner": "ann", "title": "Book dentist", "body": "", "done": true, + "created_at": "2026-01-04T15:30:00.000Z", "updated_at": "2026-01-06T08:10:00.000Z"}, + {"id": 3, "owner": "ben", "title": "Draft the quarterly report", "body": "Due Friday.", "done": false, + "created_at": "2026-01-03T11:00:00.000Z", "updated_at": "2026-01-03T11:00:00.000Z"} + ] +} diff --git a/packages/agentenv-protocol/examples/website/notes/env.toml b/packages/agentenv-protocol/examples/website/notes/env.toml new file mode 100644 index 000000000..483d79d77 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/env.toml @@ -0,0 +1,3 @@ +type = "website" +spec = "website.toml" +backend_image = { dockerfile = "Dockerfile" } diff --git a/packages/agentenv-protocol/examples/website/notes/site.py b/packages/agentenv-protocol/examples/website/notes/site.py new file mode 100644 index 000000000..777c83c40 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/site.py @@ -0,0 +1,129 @@ +"""Notes: the reference site of ``urn:agentenv:website/v1``, a small notes app on WebsiteEnvironment. Each user keeps +notes they create, edit, mark done and delete; a browser with no session is signed in as the acting user, the seed's +``current_user``, and can sign in as another seeded user. The site keeps its state in the default store, which the data +plane loads, exports and resets, and stamps each note with the universe's time. ``admin`` callers, as the gateway's +``AgentEnv-Role`` names them, can list every user's notes. + +Run it with ``python site.py`` after building ``web/`` into ``web/dist``; it serves on ``PORT`` (8000).""" +from __future__ import annotations + +from typing import Optional + +from pydantic import BaseModel + +from agentenv_protocol.website import ( + ApiContext, + BadRequest, + NotFound, + Unauthorized, + WebsiteEnvironment, + api, + ready_check, + website, +) +from agentenv_protocol.website.clock import iso + +NOTES = "notes" +USERS = "users" + + +class NewNote(BaseModel): + title: str + body: str = "" + + +class NoteEdit(BaseModel): + title: Optional[str] = None + body: Optional[str] = None + done: Optional[bool] = None + + +class SignIn(BaseModel): + user: str + + +@website(spec="website.toml") +class Notes(WebsiteEnvironment): + @ready_check + def seeded(self) -> bool: + """The site is usable once its store knows who may sign in.""" + return bool(self.store[USERS]) + + @api.get("/api/users", name="list_users") + def list_users(self) -> list: + """The users a browser can sign in as.""" + return self.store[USERS] + + @api.post("/api/sign-in", name="sign_in") + def sign_in(self, body: SignIn, ctx: ApiContext) -> dict: + """Sign the browser in as one of the site's users.""" + if self.store.get(USERS, body.user) is None: + raise NotFound(f"no user {body.user!r}") + ctx.sign_in(body.user) + return {"user": ctx.user} + + @api.get("/api/notes", name="list_notes") + def list_notes(self, ctx: ApiContext, done: Optional[bool] = None) -> list: + """The signed-in user's notes, newest first; with ``done``, only those done or not.""" + match = {"owner": self._user(ctx)} if done is None else {"owner": self._user(ctx), "done": done} + return sorted(self.store.find(NOTES, **match), key=lambda note: note["created_at"], reverse=True) + + @api.post("/api/notes", name="create_note", status=201) + def create_note(self, note: NewNote, ctx: ApiContext) -> dict: + """Add a note for the signed-in user, not done, created and updated now.""" + now = iso(self.now()) + return self.store.insert(NOTES, {"owner": self._user(ctx), "title": _title(note.title), "body": note.body, + "done": False, "created_at": now, "updated_at": now}) + + @api.get("/api/notes/{note_id:int}", name="get_note") + def get_note(self, note_id: int, ctx: ApiContext) -> dict: + return self._note(note_id, ctx) + + @api.patch("/api/notes/{note_id:int}", name="edit_note") + def edit_note(self, note_id: int, edit: NoteEdit, ctx: ApiContext) -> dict: + """Change a note's title, body or done, and its ``updated_at``.""" + note = self._note(note_id, ctx) + fields = edit.model_dump(exclude_none=True) + if "title" in fields: + fields["title"] = _title(fields["title"]) + note.update(fields, updated_at=iso(self.now())) + return note + + @api.post("/api/notes/{note_id:int}/toggle", name="toggle_note") + def toggle_note(self, note_id: int, ctx: ApiContext) -> dict: + """Mark a note done, or not done when it was.""" + note = self._note(note_id, ctx) + note.update(done=not note["done"], updated_at=iso(self.now())) + return note + + @api.delete("/api/notes/{note_id:int}", name="delete_note") + def delete_note(self, note_id: int, ctx: ApiContext) -> dict: + self.store.delete(NOTES, self._note(note_id, ctx)["id"]) + return {"deleted": note_id} + + @api.get("/api/admin/notes", name="all_notes", roles=["admin"]) + def all_notes(self) -> list: + """Every user's notes, for an ``admin`` caller only.""" + return self.store[NOTES] + + def _user(self, ctx: ApiContext) -> str: + if ctx.user is None: + raise Unauthorized("sign in first") + return ctx.user + + def _note(self, note_id: int, ctx: ApiContext) -> dict: + """The signed-in user's note ``note_id``; another user's is as missing as one that doesn't exist.""" + note = self.store.get(NOTES, note_id) + if note is None or note["owner"] != self._user(ctx): + raise NotFound(f"no note {note_id}") + return note + + +def _title(title: str) -> str: + if not title.strip(): + raise BadRequest("a note needs a title", code="empty_title") + return title.strip() + + +if __name__ == "__main__": + Notes().serve() diff --git a/packages/agentenv-protocol/examples/website/notes/web/index.html b/packages/agentenv-protocol/examples/website/notes/web/index.html new file mode 100644 index 000000000..c714b7a6c --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/web/index.html @@ -0,0 +1,37 @@ + + + + + + Notes + + + +
+

Notes

+
+ + +
+
+
+
+ + + +
+ + +
    +
    + + + diff --git a/packages/agentenv-protocol/examples/website/notes/web/package-lock.json b/packages/agentenv-protocol/examples/website/notes/web/package-lock.json new file mode 100644 index 000000000..938f9c2cd --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/web/package-lock.json @@ -0,0 +1,1020 @@ +{ + "name": "notes-site-web", + "version": "0.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "notes-site-web", + "version": "0.0.0", + "devDependencies": { + "vite": "5.4.10" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz", + "integrity": "sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.21.5.tgz", + "integrity": "sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.21.5.tgz", + "integrity": "sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.21.5.tgz", + "integrity": "sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.21.5.tgz", + "integrity": "sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.21.5.tgz", + "integrity": "sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.21.5.tgz", + "integrity": "sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.21.5.tgz", + "integrity": "sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.21.5.tgz", + "integrity": "sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.21.5.tgz", + "integrity": "sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.21.5.tgz", + "integrity": "sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.21.5.tgz", + "integrity": "sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.21.5.tgz", + "integrity": "sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.21.5.tgz", + "integrity": "sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.21.5.tgz", + "integrity": "sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.21.5.tgz", + "integrity": "sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.21.5.tgz", + "integrity": "sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz", + "integrity": "sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz", + "integrity": "sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz", + "integrity": "sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.21.5.tgz", + "integrity": "sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.21.5.tgz", + "integrity": "sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.21.5.tgz", + "integrity": "sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@napi-rs/lzma-linux-x64-gnu": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/@napi-rs/lzma-linux-x64-gnu/-/lzma-linux-x64-gnu-1.5.1.tgz", + "integrity": "sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^22.20 || ^24.12 || >=25" + } + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.67.2.tgz", + "integrity": "sha512-qB7Gzl1QI45zP1zXgqLKzjFmOBGQw2s2MLtdw0eB4xdFGPNE8LhmOnItFgyYJXXNSW8Go9c2ovvXrUUzw5sAMQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.67.2.tgz", + "integrity": "sha512-8+hXNO1J+GsYCGHwyJuBlnhyozivEvIEm/P6NkJeuoVUN+60TDvYYXk38UTm5bKRTLhIpqEQYx4d9CeN6kjh0Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-x64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-x64/-/rollup-android-x64-4.67.2.tgz", + "integrity": "sha512-/uEZY+tcWDE8DXa/rajU3Bwv6IfvQavvzjmbkn39GgQenT+UYtAxfEMdB7VKr74y8aonncm0eQa1EbiNfg+Cjg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.67.2.tgz", + "integrity": "sha512-3tCLEgakEvb5M3/KyYdhPjoteyWYziMTeznoXfcs/31eeFu+Vgs0VNRlb6Zc+YC07/sLa84OwTDF5zHOMFk92Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.67.2.tgz", + "integrity": "sha512-z31ODzkdkX7OOJCssvpz++hhtkQKWKw2L0eO2HAxFP7RMSXXedzkrTjesqCUxs4XLckRShjNfxsNxjGY5DQCGg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.67.2.tgz", + "integrity": "sha512-CTvrb0KTAQqcrpcj7rg8FgAI0+jRWnCl+En6exqXDZztCo7vspxlRUX8z9Hp/J/ja2XRnQ1JDbexkBtRyD9jSw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.67.2.tgz", + "integrity": "sha512-SJdblZpBkbOh1e4JwMM5pJJ2PdVmD3WWbMTXvncMSRLA3DdZuwU+P93RaSChOAXN4Mu53tBoFp1ADPzi3dIVAA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.67.2.tgz", + "integrity": "sha512-pUBEd7uI9dAu3TpbPcvi9eU2aJslsjIomormeTaQHmzWNhyqNx3mmq+ekC6blGNZu8F3uTHngTibKp7tbblF0A==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.67.2.tgz", + "integrity": "sha512-wW6ARw2Ah0/teOBTgTeSJIpVdPKaxQmEgZ/irkUfr3F9sgimqpApCv/NbQzjm0Nfi5EFilHh1H8kz01uQLaq5Q==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.67.2.tgz", + "integrity": "sha512-xcFYMHNL42fRu2MFipvxznr+zd+DRqwL98bhNvDWbMPRcFtMYf34NAz4xVOlnEhzgpNRE5htSemcqAsAxf2eQg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.67.2.tgz", + "integrity": "sha512-U9HrnJ9PFXqA//P/oEGaXBIJraEqzYMS9yuQOdOcPCvHZIC1FFOlnd8ocAaeDDPDpY8xOrhfxMRtr6KqKi3gyg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-gnu": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.67.2.tgz", + "integrity": "sha512-2hZN8CFGfsbDw+DNtImkPRitpLNjVM2XEbi4ZQ5igLxMFD4gI/P6Zo9czGxuvQXtRyZtZyZp8Kr4ERsAipplBw==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-musl": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.67.2.tgz", + "integrity": "sha512-HZO7UfWEsReAnkdVByyQaFsNXJCtvdfun/LFb2Xd4qnhgF1ZfhikpbgqrTRtNMmB5OgNnSwFLiPobOT8oNT9iw==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-gnu": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.67.2.tgz", + "integrity": "sha512-AdYIihuLkJ2TScy9BpSPiDex3zOMjkNJLsFlk9NeJ0U3WEtOx4W8z0nOkPEhDS2S4gwsRWBa6ZZexJ9e6OxyzQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-musl": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.67.2.tgz", + "integrity": "sha512-iZE0NXcAGqLoCWeHRPkBxJe93JT4Q981z1ZqR+y7qAiIksdRlWhw8hmqHFo1B4ZnU66He31NUb52Iyv+vwMEBQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.67.2.tgz", + "integrity": "sha512-vC7EAtJpSqSrrbg8huMfgxtwlR8fUsFCe5rStDk1WA8POky/6EEp52dxENMmqLKjSfdM3TT1WWPcZ01hiYZcdw==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.67.2.tgz", + "integrity": "sha512-DdqG6x72MHpQOlkK/cqrru9b2QMJHLkNojCB3bteB5fmkE/7XEAkReJMANp8wgef8eiEC99SvdP/BH7Ho/BpLA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.67.2.tgz", + "integrity": "sha512-F10mjdnRmGg9K+LGdqQVeUYFr08f3fVBOh7YNE5p1KIpTzhtKKOLqcQEsqZG6yEM2eouXaMo2AZzeDAcseVTrQ==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.67.2.tgz", + "integrity": "sha512-ChzK9f3iE4N5GInBquiZtzWOqh+CYS5nlLlXGJxeG3d2T9rk8L1Hxg0s+v2Eh6vmAc/1T6IhOVkQU0ORP2eV7g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.67.2.tgz", + "integrity": "sha512-95WwzC+KH9hPHtbgwvwwW6MkKME6vMAdAQLS2diCvoi/iUiYdC0Y1RIPYVESx0iWqkN5ZaZq6qD1BflLcyTMEQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-openbsd-x64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.67.2.tgz", + "integrity": "sha512-eKbpwlolQmDKd/dbFjvqB1+2qdYQLrgxi3Mtbijp7/3yBBvO9OoHtQ3ecYIaCdW46ijSjy5/yydyrrxHI416/g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ] + }, + "node_modules/@rollup/rollup-openharmony-arm64": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.67.2.tgz", + "integrity": "sha512-is9lnlv6khapl+H9Hjp0gIVsUVtDxQga2nwMHpQbhDXMCMGcNmlel4/q9ihf/3FvvbL4GdGOPOGvWoei2RCggg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.67.2.tgz", + "integrity": "sha512-yTq+7YjsgbBcMdlk7Wq+VoJilGQtdtRc4PyTvN7vhO958atCO2nQeIjCF/YANSD/Z/fnrCr27fYsGgC6PFTndQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.67.2.tgz", + "integrity": "sha512-nAvZib4Xz1zx8OKhtRuyCsSmCmbfegv1AGbfpG+CowQKxCBW1pLlfYWjcr1+GI3BzRKQbBl7l9640GwWAwOF7g==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-gnu": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.67.2.tgz", + "integrity": "sha512-R3CVbEbNa1OXjpldD1tOLarb5rB08xjQiRMpe7PZKtT8eHCZhrV4ClSPfLLbSgRMM1RfLtmAfouBRf14f47lMA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.67.2.tgz", + "integrity": "sha512-DeuVFilvaGtoS9TFIbTIQ0Kt4CBqP3xnjJ8SFqWWbXlFbrpmCdivEW7Rm38trRA8rZzQQ8lhs73nEetWotJpTg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@types/estree": { + "version": "1.0.10", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.10.tgz", + "integrity": "sha512-lTyOMnOcmRBzdCv6/4zUZCrEhV0xrqQiUZBZtVcbmLHBXKLnN8q89v/wKRF/Q0HbwVsqbIG3o5HDoFl9sduM+g==", + "dev": true, + "license": "MIT" + }, + "node_modules/esbuild": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.21.5.tgz", + "integrity": "sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=12" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.21.5", + "@esbuild/android-arm": "0.21.5", + "@esbuild/android-arm64": "0.21.5", + "@esbuild/android-x64": "0.21.5", + "@esbuild/darwin-arm64": "0.21.5", + "@esbuild/darwin-x64": "0.21.5", + "@esbuild/freebsd-arm64": "0.21.5", + "@esbuild/freebsd-x64": "0.21.5", + "@esbuild/linux-arm": "0.21.5", + "@esbuild/linux-arm64": "0.21.5", + "@esbuild/linux-ia32": "0.21.5", + "@esbuild/linux-loong64": "0.21.5", + "@esbuild/linux-mips64el": "0.21.5", + "@esbuild/linux-ppc64": "0.21.5", + "@esbuild/linux-riscv64": "0.21.5", + "@esbuild/linux-s390x": "0.21.5", + "@esbuild/linux-x64": "0.21.5", + "@esbuild/netbsd-x64": "0.21.5", + "@esbuild/openbsd-x64": "0.21.5", + "@esbuild/sunos-x64": "0.21.5", + "@esbuild/win32-arm64": "0.21.5", + "@esbuild/win32-ia32": "0.21.5", + "@esbuild/win32-x64": "0.21.5" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/nanoid": { + "version": "3.3.20", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.20.tgz", + "integrity": "sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/postcss": { + "version": "8.5.32", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.32.tgz", + "integrity": "sha512-LJlKCCAUFKOuZxipUdYw1NzIITEsPcgyOG9LkvRJvuKj/bRzbuRY0ARXvL0pWGpsZvn8jku+LMep2eU0seHp8g==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.20", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.2" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/rollup": { + "version": "4.67.2", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.67.2.tgz", + "integrity": "sha512-sFZk5BeRUQzkjmuClrJI41BowIP4yv/40d5gM9vtjxVLEt+8JG9FGlBps6cliTP2KhFiw22oqNGo0G9pVpPZlw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.10" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@napi-rs/lzma-linux-x64-gnu": "1.5.1", + "@rollup/rollup-android-arm-eabi": "4.67.2", + "@rollup/rollup-android-arm64": "4.67.2", + "@rollup/rollup-android-x64": "4.67.2", + "@rollup/rollup-darwin-arm64": "4.67.2", + "@rollup/rollup-darwin-x64": "4.67.2", + "@rollup/rollup-freebsd-arm64": "4.67.2", + "@rollup/rollup-freebsd-x64": "4.67.2", + "@rollup/rollup-linux-arm-gnueabihf": "4.67.2", + "@rollup/rollup-linux-arm-musleabihf": "4.67.2", + "@rollup/rollup-linux-arm64-gnu": "4.67.2", + "@rollup/rollup-linux-arm64-musl": "4.67.2", + "@rollup/rollup-linux-loong64-gnu": "4.67.2", + "@rollup/rollup-linux-loong64-musl": "4.67.2", + "@rollup/rollup-linux-ppc64-gnu": "4.67.2", + "@rollup/rollup-linux-ppc64-musl": "4.67.2", + "@rollup/rollup-linux-riscv64-gnu": "4.67.2", + "@rollup/rollup-linux-riscv64-musl": "4.67.2", + "@rollup/rollup-linux-s390x-gnu": "4.67.2", + "@rollup/rollup-linux-x64-gnu": "4.67.2", + "@rollup/rollup-linux-x64-musl": "4.67.2", + "@rollup/rollup-openbsd-x64": "4.67.2", + "@rollup/rollup-openharmony-arm64": "4.67.2", + "@rollup/rollup-win32-arm64-msvc": "4.67.2", + "@rollup/rollup-win32-ia32-msvc": "4.67.2", + "@rollup/rollup-win32-x64-gnu": "4.67.2", + "@rollup/rollup-win32-x64-msvc": "4.67.2", + "fsevents": "~2.3.2" + } + }, + "node_modules/source-map-js": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz", + "integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/vite": { + "version": "5.4.10", + "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.10.tgz", + "integrity": "sha512-1hvaPshuPUtxeQ0hsVH3Mud0ZanOLwVTneA1EgbAM5LhaZEqyPWGRQ7BtaMvUrTDeEaC8pxtj6a6jku3x4z6SQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.21.3", + "postcss": "^8.4.43", + "rollup": "^4.20.0" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^18.0.0 || >=20.0.0", + "less": "*", + "lightningcss": "^1.21.0", + "sass": "*", + "sass-embedded": "*", + "stylus": "*", + "sugarss": "*", + "terser": "^5.4.0" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + } + } + } + } +} diff --git a/packages/agentenv-protocol/examples/website/notes/web/package.json b/packages/agentenv-protocol/examples/website/notes/web/package.json new file mode 100644 index 000000000..c741095dd --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/web/package.json @@ -0,0 +1,14 @@ +{ + "name": "notes-site-web", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "dev": "vite", + "build": "vite build", + "preview": "vite preview" + }, + "devDependencies": { + "vite": "5.4.10" + } +} diff --git a/packages/agentenv-protocol/examples/website/notes/web/src/main.js b/packages/agentenv-protocol/examples/website/notes/web/src/main.js new file mode 100644 index 000000000..3996ac11e --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/web/src/main.js @@ -0,0 +1,231 @@ +import "./style.css"; +import { api, followUniverseClock, session, signIn, universeNow } from "./site-client.js"; + +let filter = "all"; +let renders = 0; +const saving = new Map(); +const lastSent = new Map(); +const landed = new Map(); +const drafts = new Map(); +const CLOCK_REFRESH_MS = 5000; + +const $ = (id) => document.getElementById(id); + +function setDraft(id, kind, value, saved) { + const draft = { ...drafts.get(id) }; + if (value === saved) delete draft[kind]; + else draft[kind] = value; + if (Object.keys(draft).length) drafts.set(id, draft); + else drafts.delete(id); +} + +function settleDraft(id, sent) { + const draft = drafts.get(id); + if (!draft) return; + for (const kind of Object.keys(sent)) if (draft[kind] === sent[kind]) delete draft[kind]; + if (!Object.keys(draft).length) drafts.delete(id); +} + +function queueSave(id, fields) { + lastSent.set(id, fields); + const queued = saving.get(id); + if (queued) { + queued.next = fields; + return; + } + const state = { next: fields }; + saving.set(id, state); + state.done = run(async () => { + try { + while (state.next) { + const next = state.next; + state.next = null; + await api(`api/notes/${id}`, { method: "PATCH", body: next }); + settleDraft(id, next); + } + landed.set(id, renders); + } catch (error) { + state.error = error; + lastSent.set(id, {}); + landed.delete(id); + throw error; + } finally { + saving.delete(id); + } + }); +} + +function showError(error) { + $("error").textContent = error ? error.error || error.message : ""; + $("error").hidden = !error; +} + +function formatTime(value) { + return new Date(value).toLocaleString(undefined, { dateStyle: "medium", timeStyle: "short", timeZone: "UTC" }); +} + +function tick() { + $("universe-now").textContent = `${formatTime(universeNow())} UTC`; + $("universe-now").dateTime = universeNow().toISOString(); +} + +function noteItem(note) { + const item = document.createElement("li"); + item.className = `note card${note.done ? " done" : ""}`; + item.dataset.id = note.id; + + const done = document.createElement("input"); + done.type = "checkbox"; + done.checked = note.done; + done.setAttribute("aria-label", `Mark "${note.title}" ${note.done ? "not done" : "done"}`); + done.addEventListener("change", () => run(() => api(`api/notes/${note.id}/toggle`, { method: "POST" }))); + + const draft = drafts.get(note.id) ?? {}; + const title = document.createElement("input"); + title.className = "title"; + title.defaultValue = note.title; + title.value = draft.title ?? note.title; + title.setAttribute("aria-label", "Title"); + + const body = document.createElement("textarea"); + body.className = "body"; + body.rows = 2; + body.defaultValue = note.body; + body.value = draft.body ?? note.body; + body.placeholder = "Details"; + body.setAttribute("aria-label", "Details"); + + const save = () => { + const sent = lastSent.get(note.id) ?? { title: note.title, body: note.body }; + if (title.value === sent.title && body.value === sent.body) return; + queueSave(note.id, { title: title.value, body: body.value }); + }; + for (const [kind, field] of [["title", title], ["body", body]]) { + field.addEventListener("input", () => setDraft(note.id, kind, field.value, note[kind])); + field.addEventListener("change", save); + field.addEventListener("blur", save); + } + + const meta = document.createElement("small"); + meta.textContent = `Created ${formatTime(note.created_at)} · updated ${formatTime(note.updated_at)}`; + + const remove = document.createElement("button"); + remove.className = "delete"; + remove.textContent = "Delete"; + remove.addEventListener("click", () => run(async () => { + await api(`api/notes/${note.id}`, { method: "DELETE" }); + drafts.delete(note.id); + })); + + const text = document.createElement("div"); + text.className = "text"; + text.append(title, body, meta); + item.append(done, text, remove); + return item; +} + +function focused() { + const field = document.activeElement; + if (!field?.matches?.("#notes .title, #notes .body")) return null; + return { id: field.closest("li").dataset.id, kind: field.className, start: field.selectionStart, + end: field.selectionEnd }; +} + +function refocus(kept) { + if (!kept) return; + const field = document.querySelector(`#notes li[data-id="${kept.id}"] .${kept.kind}`); + if (!field) return; + field.focus(); + field.setSelectionRange(kept.start, kept.end); +} + +async function render() { + const query = filter === "all" ? "" : `?done=${filter === "done"}`; + const request = ++renders; + const notes = await api(`api/notes${query}`); + if (request !== renders) return; + for (const [id, after] of landed) { + if (request > after && !saving.has(id)) { + lastSent.delete(id); + landed.delete(id); + } + } + const kept = focused(); + $("notes").replaceChildren(...notes.map(noteItem)); + refocus(kept); + if (!notes.length) { + const empty = document.createElement("li"); + empty.className = "empty"; + empty.textContent = "No notes here yet."; + $("notes").append(empty); + } +} + +async function run(action) { + try { + await action(); + showError(null); + await render(); + } catch (error) { + showError(error); + } +} + +async function start() { + const [signedIn, users] = await Promise.all([session(), api("api/users")]); + $("user").replaceChildren(...users.map((user) => new Option(user.name || user.id, user.id))); + let signedInAs = signedIn.user ?? ""; + $("user").value = signedInAs; + $("user").addEventListener("change", (event) => { + const user = event.target.value; + $("user").disabled = true; + run(async () => { + try { + while (saving.size) { + const pending = [...saving.values()]; + await Promise.all(pending.map((state) => state.done)); + const failed = pending.find((state) => state.error); + if (failed) throw failed.error; + } + await signIn(user); + signedInAs = user; + drafts.clear(); + } finally { + $("user").value = signedInAs; + $("user").disabled = false; + } + }); + }); + + const add = $("new-note").querySelector("button[type=submit]"); + $("new-note").addEventListener("submit", (event) => { + event.preventDefault(); + if (add.disabled) return; + const body = { title: $("new-title").value, body: $("new-body").value }; + add.disabled = true; + run(async () => { + try { + await api("api/notes", { method: "POST", body }); + } finally { + add.disabled = false; + } + if ($("new-title").value === body.title) $("new-title").value = ""; + if ($("new-body").value === body.body) $("new-body").value = ""; + }); + }); + + for (const button of document.querySelectorAll("[data-filter]")) { + button.addEventListener("click", () => { + filter = button.dataset.filter; + document.querySelectorAll("[data-filter]").forEach((b) => b.classList.toggle("active", b === button)); + run(async () => {}); + }); + } + + tick(); + setInterval(tick, 1000); + followUniverseClock({ path: "healthz", everyMs: CLOCK_REFRESH_MS }); + await render(); +} + +start().catch(showError); diff --git a/packages/agentenv-protocol/examples/website/notes/web/src/site-client.js b/packages/agentenv-protocol/examples/website/notes/web/src/site-client.js new file mode 100644 index 000000000..853cb12c4 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/web/src/site-client.js @@ -0,0 +1,89 @@ +// The browser client of a site on agentenv_protocol.website: JSON calls to the site's API, the universe's time from +// X-Universe-Now, and the signed-in user. It has no dependencies. Paths are relative to the page, so a site works at +// "/" and under the gateway's "/website//" alike: a leading "/" is dropped. + +let offsetMs = 0; + +// An answer other than success: `status` (0 when the server couldn't be reached), and the site's `code` and `error`. +export class ApiError extends Error { + constructor(status, code, error, body) { + super(error); + this.name = "ApiError"; + this.status = status; + this.code = code; + this.error = error; + this.body = body; + } +} + +// The universe's time: the X-Universe-Now of the latest answer, extrapolated at real speed since it came. An idle +// page keeps it right with followUniverseClock(). +export function universeNow() { + return new Date(Date.now() + offsetMs); +} + +// Read X-Universe-Now from `path`, the site's health path, every `everyMs`, so universeNow() follows a clock that +// stops or runs at another speed while the page makes no calls. Returns a function that stops it. +export function followUniverseClock({ path = "api/health", everyMs = 5000 } = {}) { + const id = setInterval(() => fetch(relative(path), { credentials: "same-origin" }).then(follow).catch(() => {}), everyMs); + return () => clearInterval(id); +} + +function follow(response) { + const at = Date.parse(response.headers.get("X-Universe-Now") || ""); + if (!Number.isNaN(at)) offsetMs = at - Date.now(); +} + +function relative(path) { + return String(path).replace(/^\/+/, ""); +} + +// Call the site's API: `body` is sent as JSON; the answer's JSON is returned, or an ApiError thrown. +export async function api(path, { method = "GET", body, headers = {} } = {}) { + const url = relative(path); + let response; + try { + response = await fetch(url, { + method, + credentials: "same-origin", + headers: body === undefined ? headers : { "Content-Type": "application/json", ...headers }, + body: body === undefined ? undefined : JSON.stringify(body), + }); + } catch (cause) { + throw new ApiError(0, "network_error", `${method} ${url}: ${cause?.message || cause}`, null); + } + follow(response); + let text; + try { + text = await response.text(); + } catch (cause) { + throw new ApiError(response.status, "network_error", `${method} ${url}: ${cause?.message || cause}`, null); + } + let data = null; + try { + data = text ? JSON.parse(text) : null; + } catch { + data = text; + } + if (!response.ok) { + const error = data && typeof data === "object" && data.error ? data.error : `${method} ${url}: ${response.status}`; + const code = data && typeof data === "object" && data.code ? data.code : "http_error"; + throw new ApiError(response.status, code, error, data); + } + return data; +} + +// Who the browser is signed in as: {user, role, signed_in}. A browser with no session is signed in as the site's +// acting user. +export function session() { + return api("api/session"); +} + +// Sign in as `user` through the site's own sign-in route, `api/sign-in` unless `path` names another. +export function signIn(user, { path = "api/sign-in" } = {}) { + return api(path, { method: "POST", body: { user } }); +} + +export function signOut() { + return api("api/session", { method: "DELETE" }); +} diff --git a/packages/agentenv-protocol/examples/website/notes/web/src/style.css b/packages/agentenv-protocol/examples/website/notes/web/src/style.css new file mode 100644 index 000000000..8ecd48b6a --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/web/src/style.css @@ -0,0 +1,57 @@ +:root { + --bg: #f6f6f3; + --card: #ffffff; + --text: #1d1d1b; + --muted: #6b6b66; + --line: #deded8; + --accent: #2f5bd3; + font-family: system-ui, -apple-system, "Segoe UI", sans-serif; + color: var(--text); + background: var(--bg); +} + +* { box-sizing: border-box; } +body { margin: 0; } + +.bar { + display: flex; + flex-wrap: wrap; + gap: 12px; + align-items: center; + justify-content: space-between; + padding: 12px 16px; + border-bottom: 1px solid var(--line); + background: var(--card); +} +.bar h1 { margin: 0; font-size: 1.25rem; } +.who { display: flex; flex-wrap: wrap; gap: 12px; align-items: center; color: var(--muted); font-size: 0.9rem; } +.who time { font-variant-numeric: tabular-nums; } + +main { max-width: 720px; margin: 0 auto; padding: 16px; } + +.card { background: var(--card); border: 1px solid var(--line); border-radius: 8px; padding: 12px; } + +input, textarea, select, button { font: inherit; color: inherit; } +input, textarea { width: 100%; border: 1px solid var(--line); border-radius: 6px; padding: 6px 8px; background: transparent; } +textarea { resize: vertical; } +button { border: 1px solid var(--line); border-radius: 6px; padding: 6px 12px; background: var(--card); cursor: pointer; } +button[type="submit"], .filters .active { background: var(--accent); border-color: var(--accent); color: #fff; } + +#new-note { display: grid; gap: 8px; } +#new-note button { justify-self: end; } + +.filters { display: flex; gap: 8px; margin: 16px 0; } + +#error { color: #b3261e; margin: 0 0 12px; } + +#notes { list-style: none; margin: 0; padding: 0; display: grid; gap: 8px; } +.note { display: grid; grid-template-columns: auto 1fr auto; gap: 12px; align-items: start; } +.note .text { display: grid; gap: 6px; min-width: 0; } +.note .title { font-weight: 600; border-color: transparent; } +.note .body { border-color: transparent; color: var(--muted); } +.note .title:focus, .note .body:focus { border-color: var(--line); } +.note small { color: var(--muted); } +.note.done .title { text-decoration: line-through; color: var(--muted); } +.note input[type="checkbox"] { width: auto; margin-top: 10px; } +.delete { color: #b3261e; } +.empty { color: var(--muted); text-align: center; padding: 24px; } diff --git a/packages/agentenv-protocol/examples/website/notes/web/vite.config.js b/packages/agentenv-protocol/examples/website/notes/web/vite.config.js new file mode 100644 index 000000000..d07ecd2c2 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/web/vite.config.js @@ -0,0 +1,11 @@ +import { defineConfig } from "vite"; + +export default defineConfig({ + base: "./", + server: { + proxy: { + "/api": "http://127.0.0.1:8000", + "/healthz": "http://127.0.0.1:8000", + }, + }, +}); diff --git a/packages/agentenv-protocol/examples/website/notes/website.toml b/packages/agentenv-protocol/examples/website/notes/website.toml new file mode 100644 index 000000000..e0bb05473 --- /dev/null +++ b/packages/agentenv-protocol/examples/website/notes/website.toml @@ -0,0 +1,38 @@ +[site] +name = "notes" +domains = ["notes.test"] +health = "/healthz" + +[source] +sdk = "site:Notes" +assets = "web/dist" + +[data] +seed = ["data/seed.json"] + +[data.collections.users] + +[data.collections.notes] +owner = "owner" + +[users] +personas = [{ id = "ann", name = "Ann Lee" }, { id = "ben", name = "Ben Ortiz" }] +acting_user = "ann" +sign_in = "cookie" + +[roles] +admin = ["all_notes"] + +[routes.ops] +list_users = "GET /api/users" +sign_in = "POST /api/sign-in" +list_notes = "GET /api/notes" +create_note = "POST /api/notes" +get_note = "GET /api/notes/{note_id}" +edit_note = "PATCH /api/notes/{note_id}" +toggle_note = "POST /api/notes/{note_id}/toggle" +delete_note = "DELETE /api/notes/{note_id}" +all_notes = "GET /api/admin/notes" + +[clock] +follow = "header" diff --git a/packages/agentenv-protocol/examples/website/replay/Dockerfile b/packages/agentenv-protocol/examples/website/replay/Dockerfile new file mode 100644 index 000000000..c641d704b --- /dev/null +++ b/packages/agentenv-protocol/examples/website/replay/Dockerfile @@ -0,0 +1,14 @@ +ARG REPLAY=packages/agentenv-protocol/examples/website/replay + +FROM mirror.gcr.io/library/python:3.12-slim@sha256:423ed6ab25b1921a477529254bfeeabf5855151dc2c3141699a1bfc852199fbf +ARG REPLAY +WORKDIR /app +RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/* +COPY packages/agentenv-protocol /tmp/agentenv-protocol +RUN pip install --no-cache-dir /tmp/agentenv-protocol "uvicorn>=0.30" && rm -rf /tmp/agentenv-protocol +COPY ${REPLAY}/site.py ${REPLAY}/capture.warc.gz /app/ +ENV PORT=8000 +EXPOSE 8000 +RUN useradd --create-home --uid 10001 appuser +USER appuser +CMD ["python", "site.py"] diff --git a/packages/agentenv-protocol/examples/website/replay/Dockerfile.dockerignore b/packages/agentenv-protocol/examples/website/replay/Dockerfile.dockerignore new file mode 100644 index 000000000..10ae9270c --- /dev/null +++ b/packages/agentenv-protocol/examples/website/replay/Dockerfile.dockerignore @@ -0,0 +1,6 @@ +**/node_modules +**/dist +**/__pycache__ +**/.pytest_cache +**/.venv +.git diff --git a/packages/agentenv-protocol/examples/website/replay/README.md b/packages/agentenv-protocol/examples/website/replay/README.md new file mode 100644 index 000000000..b6c48e3ba --- /dev/null +++ b/packages/agentenv-protocol/examples/website/replay/README.md @@ -0,0 +1,77 @@ +# Widget Docs: a captured site, replayed + +Widget Docs is a small docs site captured to a WARC file and served by +`agentenv_protocol.website.replay.ReplayWebsite`. The captured pages are served as they were captured, for any host, +with only their same-site links rewritten to stay on the replay. The newsletter sign-up, which a capture can't answer, is written as `@api` routes on top. + +| File | What it is | +| --- | --- | +| `source/` | The site that was captured: a home page, two guide pages, a stylesheet and a script. | +| `capture.warc.gz` | The capture of `source/` at `--depth 1`: `/`, `/guide/install.html`, `/guide/usage.html?utm_source=home`, `/style.css` and `/app.js`. `/guide/advanced.html` is two links from home, so it isn't in the capture. | +| `Dockerfile`, `env.toml` | The server image, built from the repo root as the Notes example's is, and the website env as a bundle entry. | +| `site.py` | The server: `WidgetDocs(ReplayWebsite)` with `capture = "capture.warc.gz"`, plus `POST /api/subscribe` and `GET /api/subscriptions`. | + +## Capture a site + +```bash +python -m agentenv_protocol.website.replay capture https://docs.example.com/ --out capture.warc.gz --depth 2 +``` + +The tool needs only this package and `httpx`. It fetches each start URL. It then fetches the same-site pages those URLs +link to, up to `--depth` links away, and the assets (stylesheets, scripts, images) each captured page loads, with the +stylesheets, images and fonts a captured stylesheet's `@import` and `url()` name. Every +answer goes into a WARC 1.1 file, gzipped one record at a time, which pywb and warcio also read. Notes: + +- Only GETs are captured. +- Redirects are kept as they were answered, and their targets are captured too. +- Bodies are stored decoded, so the stored headers leave out `Content-Encoding` and `Transfer-Encoding`. +- A host is the same site as a start URL when it is that URL's host (without `www.`) or a subdomain of it. +- Request records name only the method, the target and `Host`, so no credential the capturing client sent ends up in + the file. +- `--max-requests` (500 by default) caps a capture, and `--no-assets` leaves out what pages load. + +To capture `source/` again: + +```bash +python -m http.server -d source 8765 & +python -m agentenv_protocol.website.replay capture http://127.0.0.1:8765/ --out capture.warc.gz --depth 1 +``` + +## How a request is matched + +A request is matched by its method (HEAD as GET), its path and its query, never by its host or headers. The query is +normalised before matching: + +- Its parameters are sorted. +- Blank parameters are kept. +- The `utm_*`, `fbclid`, `gclid`, `msclkid` and `_` parameters are dropped. Set `ignore_query` on the class to change + this list. + +Each path segment is percent-decoded and then encoded again, so `%7E` and `~` match. A request the capture doesn't +hold gets a 404. If a capture holds one request twice, the first answer is served. + +When the capture's answers are served: + +- `Set-Cookie` loses its `Domain` attribute, so the cookie works on the replay's own host. +- A `Location` that points at a captured host is made relative. +- In HTML pages (`href`, `src`, `srcset`, `action`, `formaction`, `poster`, `style` and `' + b'
    ' + b''), + "/t.css": (200, "text/css", b"p{}"), + "/h.png": (200, "image/png", b"\x89PNG"), + "/d.png": (200, "image/png", b"\x89PNG"), + "/p.jpg": (200, "image/jpeg", b"\xff\xd8"), + } + + +def test_a_capture_loads_what_a_pages_style_values_style_elements_and_posters_name(): + site = _InlineStyledSite() + try: + out = io.BytesIO() + result = capture([site.url + "/"], out, depth=0) + assert {key[1] for key in Archive.read(out.getvalue()).entries} == {"/", "/t.css", "/h.png", "/d.png", "/p.jpg"} + assert result.captured == 5 and not result.failed + finally: + site.close() + + +def test_an_imported_warcs_compressed_page_is_read_decoded_so_its_links_stay_on_the_replay(tmp_path): + page = b'a' + css = b"b{background:url(/b.png)}" + records = b"" + for url, kind, body, coding in (("https://shop.example/", "text/html", gzip.compress(page), "gzip"), + ("https://shop.example/s.css", "text/css", zlib.compress(css), "deflate")): + http = (f"HTTP/1.1 200 OK\r\nContent-Type: {kind}\r\nContent-Encoding: {coding}\r\n" + f"Content-Length: {len(body)}\r\n\r\n").encode() + body + records += (f"WARC/1.0\r\nWARC-Type: response\r\nWARC-Target-URI: {url}\r\n" + f"Content-Type: application/http;msgtype=response\r\nContent-Length: {len(http)}\r\n\r\n" + ).encode() + http + b"\r\n\r\n" + (tmp_path / "other.warc").write_bytes(records) + site = TestClient(Shop(tmp_path / "other.warc").create_app()) + behind = {"X-Forwarded-Prefix": "/website/shop"} + served = site.get("/", headers=behind) + assert "content-encoding" not in served.headers + assert served.content == b'a' + assert site.get("/s.css", headers=behind).content == b"b{background:url(/website/shop/b.png)}" + + +def test_an_imported_answer_in_a_coding_the_replay_cant_undo_is_served_as_captured(): + http = b"HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Encoding: br\r\n\r\n\x0b\x02\x80" + record = (f"WARC/1.0\r\nWARC-Type: response\r\nWARC-Target-URI: https://shop.example/\r\n" + f"Content-Type: application/http;msgtype=response\r\nContent-Length: {len(http)}\r\n\r\n").encode() + entry = Archive.read(record + http + b"\r\n\r\n").find("GET", "/") + assert entry.body == b"\x0b\x02\x80" and ("Content-Encoding", "br") in entry.headers + + +def test_a_replay_drops_a_link_tags_integrity_so_its_rewritten_stylesheet_still_loads(): + page = (b'' + b'') + site = TestClient(Shop(warc(("https://shop.example/", 200, [("Content-Type", "text/html")], page))).create_app()) + assert site.get("/").content == (b'' + b'') + + +def test_a_replay_serves_no_captured_content_security_policy_so_its_rewritten_page_still_loads(): + page = (b'' + b'') + headers = [("Content-Type", "text/html"), ("Content-Security-Policy", "style-src 'sha256-abc'"), + ("Content-Security-Policy-Report-Only", "default-src https://shop.example"), ("X-Kept", "1")] + site = TestClient(Shop(warc(("https://shop.example/", 200, headers, page))).create_app()) + served = site.get("/", headers={"X-Forwarded-Prefix": "/website/shop"}) + assert "content-security-policy" not in served.headers + assert "content-security-policy-report-only" not in served.headers + assert served.headers["x-kept"] == "1" + assert served.content == b'' + + +def test_a_replay_keeps_a_pages_and_a_stylesheets_same_site_links_on_the_replay(): + page = ('' + "" + "ahome" + 'awayctop' + '/not-a-link' + '
    ' + '
    é
    ').encode() + css = b"@import '/t.css'; a{background:url(https://shop.example/bg.png)} b{src:url(f.woff2)}" + site = TestClient(Shop(warc(("https://shop.example/", 200, [("Content-Type", "text/html; charset=utf-8")], page), + ("https://shop.example/style.css", 200, [("Content-Type", "text/css")], css), + ("https://shop.example/app.js", 200, [("Content-Type", "text/javascript")], + b"fetch('/api/x')"))).create_app()) + behind = {"X-Forwarded-Prefix": "/website/shop"} + served = site.get("/", headers=behind) + assert served.content == page.replace(b'"/style.css"', b'"/website/shop/style.css"').replace( + b'url("/b.png")', b'url("/website/shop/b.png")').replace( + b"'https://shop.example/a?x=1&y=2'", b"'/website/shop/a?x=1&y=2'").replace( + b"href=//shop.example>", b"href=/website/shop/>").replace(b"src=/i.png", b"src=/website/shop/i.png").replace( + b'"/i.png 1x, https://shop.example/i2.png 2x"', b'"/website/shop/i.png 1x, /website/shop/i2.png 2x"').replace( + b'"/search"', b'"/website/shop/search"').replace(b'"/go2"', b'"/website/shop/go2"').replace( + b"url('/d.png')", b"url('/website/shop/d.png')") + assert b"var u = '/api/x'; location.href = \"https://shop.example/js\"" in served.content + assert served.headers["content-length"] == str(len(served.content)) + assert site.head("/", headers=behind).headers["content-length"] == str(len(served.content)) + assert site.get("/style.css", headers=behind).content == ( + b"@import '/website/shop/t.css'; a{background:url(/website/shop/bg.png)} b{src:url(f.woff2)}") + assert site.get("/app.js", headers=behind).content == b"fetch('/api/x')" + + direct = site.get("/").content + assert b"'/a?x=1&y=2'" in direct and b'"/style.css"' in direct and b"https://elsewhere.example/z" in direct diff --git a/packages/agentenv-protocol/tests/test_website_spec.py b/packages/agentenv-protocol/tests/test_website_spec.py new file mode 100644 index 000000000..3fa8e7bae --- /dev/null +++ b/packages/agentenv-protocol/tests/test_website_spec.py @@ -0,0 +1,955 @@ +"""The website spec: loading and validating one, ``extends`` overlays, ``uses`` composition and its conflicts, a multi +env's name table, the card round trip, a spec derived from a WebsiteEnvironment class and a class declared from one, +and the conformance checker run against a spec.""" + +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path + +import httpx +import pytest +from starlette.testclient import TestClient + +from agentenv_protocol import environment_card +from agentenv_protocol.website import WebsiteEnvironment, api, conformance, website +from agentenv_protocol.website import spec as website_spec +from agentenv_protocol.website.spec import SpecError, from_card, from_environment, load, name_table, resolve + +EXAMPLES = Path(__file__).resolve().parents[1] / "examples" / "website" +NOTES_DIR = EXAMPLES / "notes" +SPECS_DIR = EXAMPLES / "specs" + + +def _notes_module(): + spec = importlib.util.spec_from_file_location("notes_spec_example_site", NOTES_DIR / "site.py") + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +notes_site = _notes_module() + + +def write(folder: Path, files: dict[str, str]) -> Path: + for name, text in files.items(): + (folder / name).parent.mkdir(parents=True, exist_ok=True) + (folder / name).write_text(text) + return folder / next(iter(files)) + + +def problems(path_or_raw, base_dir: Path | None = None) -> list[str]: + with pytest.raises(SpecError) as caught: + load(path_or_raw) if base_dir is None else resolve(path_or_raw, base_dir) + return caught.value.problems + + +def card_of(client: TestClient) -> dict: + return client.get("/.well-known/agent-env.json").json() + + +def website_params(card: dict) -> dict: + params = next(e for e in card["capabilities"]["extensions"] if e["uri"] == website_spec.WEBSITE_EXTENSION_URI)["params"] + return {k: v for k, v in params.items() if k != "methods"} + + +@pytest.fixture +def notes_client(tmp_path: Path) -> TestClient: + (tmp_path / "index.html").write_text("") + spec = notes_site.Notes.declaration().spec.overlay({"source": {"sdk": "site:Notes", "assets": str(tmp_path)}}) + return TestClient(website(spec=spec)(type("Notes", (notes_site.Notes,), {}))().create_app()) + + +# Loading + + +def test_the_notes_spec_loads_with_its_routes_roles_personas_and_data() -> None: + spec = load(NOTES_DIR / "website.toml") + assert (spec.site.name, spec.site.domains, spec.health) == ("notes", ("notes.test",), "/healthz") + assert (spec.source.kind, spec.source.sdk, spec.path(spec.source.assets)) == ("sdk", "site:Notes", + NOTES_DIR / "web" / "dist") + assert [p.id for p in spec.users.personas] == ["ann", "ben"] and spec.users.acting_user == "ann" + assert spec.data.collections["notes"].owner == "owner" and spec.data.seed == ("data/seed.json",) + table = spec.route_table() + assert list(table)[:3] == ["list_users", "sign_in", "list_notes"] + assert (table["get_note"].method, table["get_note"].path) == ("GET", "/api/notes/{note_id}") + assert table["all_notes"].roles == ("admin",) and table["list_notes"].roles == () + assert spec.clock.follow == "header" + + +def test_every_section_is_optional_and_a_route_takes_a_table_or_a_shorthand(tmp_path: Path) -> None: + assert load(write(tmp_path, {"empty.toml": ""})).to_card_extension() == { + "domains": [], "health": "/api/health", "routes": [], "session_cookie": "agentenv_session", + "session_path": "/api/session"} + spec = load(write(tmp_path, {"r.toml": '[routes.ops]\na = "get /x/{id:int}"\n' + 'b = { method = "post", path = "/x", roles = ["admin"] }\n'})) + assert [r.advertised(n) for n, r in spec.route_table().items()] == [ + {"method": "GET", "path": "/x/{id}", "name": "a", "roles": None}, + {"method": "POST", "path": "/x", "name": "b", "roles": ["admin"]}] + + +def test_routes_come_from_an_openapi_document_by_operation_id(tmp_path: Path) -> None: + write(tmp_path, {"openapi.json": json.dumps({"paths": { + "/api/items": {"get": {"operationId": "list_items"}, "post": {"operationId": "add_item", + "x-agentenv-roles": ["admin"]}}}})}) + spec = load(write(tmp_path, {"s.toml": '[routes]\nopenapi = "openapi.json"\n[routes.ops]\nping = "GET /ping"\n'})) + assert {n: (r.method, r.path, r.roles) for n, r in spec.route_table().items()} == { + "list_items": ("GET", "/api/items", ()), "add_item": ("POST", "/api/items", ("admin",)), + "ping": ("GET", "/ping", ())} + clash = write(tmp_path, {"c.toml": '[routes]\nopenapi = "openapi.json"\n[routes.ops]\nlist_items = "GET /other"\n'}) + assert any("list_items: GET /other isn't the openapi document's GET /api/items" in p for p in problems(clash)) + + +def test_an_openapi_document_of_another_shape_or_with_a_name_twice_is_refused(tmp_path: Path) -> None: + def found(document) -> list[str]: + write(tmp_path, {"openapi.json": json.dumps(document)}) + return problems(write(tmp_path, {"s.toml": '[routes]\nopenapi = "openapi.json"\n'})) + + shape = "openapi.json isn't an OpenAPI document, an object whose paths are an object" + assert any(shape in p for p in found([])) + assert any(shape in p for p in found({"paths": ["/a"]})) + assert any("path /a isn't an object of operations" in p for p in found({"paths": {"/a": ["get"]}})) + assert any("routes.openapi: items names both GET /a and GET /b" in p for p in found( + {"paths": {"/a": {"get": {"operationId": "items"}}, "/b": {"get": {"operationId": "items"}}}})) + + +def test_false_removes_an_operation_of_the_openapi_document(tmp_path: Path) -> None: + write(tmp_path, {"openapi.json": json.dumps({"paths": {"/api/items": { + "get": {"operationId": "list_items"}, "post": {"operationId": "add_item"}}}}), + "base.toml": '[routes]\nopenapi = "openapi.json"\n'}) + spec = load(tmp_path / "base.toml") + assert set(spec.route_table()) == {"list_items", "add_item"} and spec.routes.openapi is None + assert set(spec.overlay({"routes": {"ops": {"add_item": False}}}).route_table()) == {"list_items"} + child = load(write(tmp_path, {"child.toml": 'extends = "base.toml"\n[routes.ops]\nadd_item = false\n'})) + assert set(child.route_table()) == {"list_items"} + assert set(resolve(child.to_dict(), tmp_path).route_table()) == {"list_items"} + + +@pytest.mark.parametrize("toml, problem", [ + ("[site]\ncolor = 1\n", "site.color: isn't a key the spec takes"), + ("[pages]\n", "pages: isn't a key the spec takes"), + ('[source]\nsdk = "a:B"\nstatic = "dist"\n', "source: names exactly one of image, sdk, static, live, replay, " + "not ['sdk', 'static']"), + ("[source]\n", "source: names exactly one of image, sdk, static, live, replay, not none"), + ('[source]\nstatic = "dist"\nassets = "x"\n', "source: assets is the static folder of an sdk source"), + ('[source]\nsdk = "nocolon"\n', "source: sdk is module:Class, not 'nocolon'"), + ('[site]\ndomains = ["ok.test", "Bad.test", "a.*.test"]\n', "site.domains: ['Bad.test', 'a.*.test'] aren't host"), + ('[site]\nhealth = "/agentenv"\n', "site.health: health is a path of its own under '/', not '/agentenv'"), + ('[source.live]\ncredentials = { token = "xoxp-123" }\n', "source.live.credentials: credentials ['token'] must be"), + ('[users]\npersonas = ["ann"]\nacting_user = "zed"\n', "users: acting_user 'zed' is none of the personas ['ann']"), + ('[users]\npersonas = ["ann", "ann"]\n', "users: personas ['ann'] are declared twice"), + ('[users]\nsign_in = "sso"\n', "users.sign_in: Input should be 'cookie', 'none' or 'live'"), + ('[routes.ops]\na = "FETCH /x"\n', "routes.ops.a: method is one of GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, not 'FETCH'"), + ('[routes.ops]\na = "GET /agentenv/x"\n', "routes.ops.a: path '/agentenv/x' is the data plane's or the card's"), + ('[routes.ops]\na = "GET /x"\nb = "get /x"\n', "routes.ops: b and a are both GET /x"), + ('[site]\nhealth = "/up"\n[routes.ops]\nup = "GET /up"\n', "site.health: /up is served by route up"), + ('[roles]\nadmin = ["nope"]\n', "roles.admin: names no route ['nope']"), + ('[source]\nsdk = "a:B"\n[users]\nsign_in = "live"\n', "users.sign_in: live is the sign-in of a live or replayed"), + ('[source]\nsdk = "a:B"\n[clock]\nfollow = "none"\n', "clock.follow: an sdk site follows the universe's clock"), + ('[source]\nstatic = "dist"\n[routes.ops]\na = "GET /a"\n', "routes: a static site serves no routes"), + ('[source]\nsdk = "a:B"\n[routes.ops]\na = { method = "GET", path = "/a", served_by = "x:Y" }\n', + "routes.ops: served_by swaps a route of a live or replayed site"), + ('[data]\nschema = "s.json"\nmodels = "mcp:mail"\n', "data: data takes a schema or models, not both"), + ('[data]\nmodels = "mail"\n', "data.models: models is mcp:, not 'mail'"), + ('[users]\nsign_in = "none"\n[data.collections.notes]\nowner = "owner"\n', + "data.collections.notes: owner names a user, and the site signs nobody in"), + ('[[uses]]\nspec = "c.toml"\nmount = "/.well-known/agent-env.json"\n', "uses[0].mount: mount is a plain path"), + ("[site\n", "isn't TOML"), +], ids=lambda value: value[:24] if isinstance(value, str) else value) +def test_a_spec_that_does_not_hold_together_is_refused_naming_where(tmp_path: Path, toml: str, problem: str) -> None: + found = problems(write(tmp_path, {"s.toml": toml})) + assert any(problem in p for p in found), found + assert all(p.startswith(str(tmp_path / "s.toml")) for p in found) + + +def test_a_missing_spec_file_is_a_problem(tmp_path: Path) -> None: + assert "can't be read" in problems(tmp_path / "nope.toml")[0] + + +def test_to_dict_reads_back_as_the_same_spec() -> None: + spec = load(NOTES_DIR / "website.toml") + again = resolve(spec.to_dict(), NOTES_DIR) + assert again == spec and again.to_card_extension() == spec.to_card_extension() + + +def test_to_dict_and_an_overlay_keep_a_json_schema_as_it_is(tmp_path: Path) -> None: + closed = {"type": "object", "additionalProperties": False, "properties": {"id": {}}, "required": []} + spec = resolve({"data": {"schema": {"properties": {}}, "collections": {"items": {"schema": closed}, + "any": {"schema": {}}}}}, tmp_path) + data = spec.to_dict()["data"] + assert (data["schema"], data["collections"]["items"]["schema"], data["collections"]["any"]["schema"]) == ( + {"properties": {}}, closed, {}) + assert resolve(spec.to_dict(), tmp_path) == spec + assert spec.overlay({"site": {"name": "x"}}).data.collections["items"].schema_ == closed + + +# extends + + +BASE = """ +[site] +name = "shop" +domains = ["shop.test", "www.shop.test"] +health = "/healthz" + +[source] +sdk = "shop:Shop" +assets = "dist" + +[data] +seed = ["seed.json"] + +[data.collections.orders] +owner = "buyer" + +[data.collections.carts] + +[users] +personas = ["ann", "ben"] +acting_user = "ann" + +[roles] +admin = ["refund"] +staff = ["refund"] + +[routes.ops] +list_orders = "GET /api/orders" +refund = "POST /api/orders/{id}/refund" +""" + + +def test_an_overlay_changes_one_key_of_an_operation_of_the_openapi_document(tmp_path: Path) -> None: + write(tmp_path, {"openapi.json": json.dumps({"paths": {"/api/items": { + "get": {"operationId": "list_items"}, "post": {"operationId": "add_item", "x-agentenv-roles": ["admin"]}}}}), + "base.toml": '[routes]\nopenapi = "openapi.json"\n'}) + child = load(write(tmp_path, {"child.toml": 'extends = "base.toml"\n[source.live]\n[routes.ops]\n' + 'add_item = { roles = ["staff"] }\n' + 'list_items = { served_by = "shop:Shop" }\n'})) + routes = child.route_table() + assert (routes["add_item"].method, routes["add_item"].path, routes["add_item"].roles) == ( + "POST", "/api/items", ("staff",)) + assert (routes["list_items"].method, routes["list_items"].path, routes["list_items"].served_by) == ( + "GET", "/api/items", "shop:Shop") + assert load(tmp_path / "base.toml").overlay({"routes": {"ops": {"add_item": {"roles": ["staff"]}}}}).route_table()[ + "add_item"].roles == ("staff",) + moved = write(tmp_path, {"moved.toml": 'extends = "base.toml"\n[routes.ops]\nadd_item = { path = "/api/other" }\n'}) + assert any("add_item: POST /api/other isn't the openapi document's POST /api/items" in p for p in problems(moved)) + + +def test_extends_merges_tables_replaces_arrays_and_source_and_false_removes(tmp_path: Path) -> None: + write(tmp_path, {"base/shop.toml": BASE}) + child = write(tmp_path, {"variants/ben.toml": """ +extends = "../base/shop.toml" + +[site] +domains = ["shop.example"] + +[source.image] +dockerfile = "Dockerfile" + +[data.collections] +orders = false + +[data.collections.carts] +key = "cart_id" + +[users] +personas = ["ben", "cy"] +acting_user = "ben" + +[roles] +staff = false + +[routes.ops] +refund = { roles = ["manager"] } +list_orders = false +place_order = "POST /api/orders" +"""}) + spec = load(child) + assert (spec.site.name, spec.site.domains, spec.health) == ("shop", ("shop.example",), "/healthz") + assert spec.source.kind == "image" and spec.source.assets is None + assert spec.source.image.dockerfile == "Dockerfile" and spec.base_dir == (tmp_path / "variants").resolve() + assert spec.path(spec.data.seed[0]).resolve() == (tmp_path / "base" / "seed.json").resolve() + assert list(spec.data.collections) == ["carts"] and spec.data.collections["carts"].key == "cart_id" + assert [p.id for p in spec.users.personas] == ["ben", "cy"] and spec.users.acting_user == "ben" + assert list(spec.roles) == ["admin"] + assert {n: (r.method, r.path, r.roles) for n, r in spec.route_table().items()} == { + "refund": ("POST", "/api/orders/{id}/refund", ("manager", "admin")), + "place_order": ("POST", "/api/orders", ())} + assert spec.extends is None + + +def test_an_inherited_image_keeps_its_default_context_in_its_own_folder(tmp_path: Path) -> None: + write(tmp_path, {"base/site.toml": '[site]\nname = "shop"\n[source.image]\ndockerfile = "Dockerfile"\n'}) + implicit = load(write(tmp_path, {"variants/ben.toml": 'extends = "../base/site.toml"\n'})) + write(tmp_path, {"base/explicit.toml": '[site]\nname = "shop"\n[source.image]\ndockerfile = "Dockerfile"\n' + 'context = "."\n'}) + explicit = load(write(tmp_path, {"variants/cy.toml": 'extends = "../base/explicit.toml"\n'})) + assert implicit.path(implicit.source.image.context).resolve() == (tmp_path / "base").resolve() + assert implicit.source.image == explicit.source.image + + +def test_an_extends_cycle_is_refused(tmp_path: Path) -> None: + write(tmp_path, {"a.toml": 'extends = "b.toml"\n', "b.toml": 'extends = "a.toml"\n'}) + assert "is a cycle" in problems(tmp_path / "a.toml")[0] + + +@pytest.mark.parametrize(("files", "problem"), [ + ({"s.toml": 'extends = "base.toml"\nuses = 1\n', "base.toml": '[[uses]]\nspec = "c.toml"\n'}, "uses: "), + ({"s.toml": 'extends = "base/base.toml"\n', "base/base.toml": "[data]\ncollections = [1]\n"}, "data.collections: "), +], ids=["uses", "collections"]) +def test_an_extends_whose_fields_have_the_wrong_type_is_refused_naming_them(tmp_path: Path, files: dict, + problem: str) -> None: + found = problems(write(tmp_path, files)) + assert any(problem in p for p in found), found + + +def test_overlay_merges_over_a_loaded_spec_as_extends_does(tmp_path: Path) -> None: + spec = load(write(tmp_path, {"shop.toml": BASE})) + changed = spec.overlay({"users": {"acting_user": "ben"}, "routes": {"ops": {"refund": {"roles": []}}}}) + assert changed.users.acting_user == "ben" and changed.route_table()["refund"].roles == ("admin", "staff") + assert spec.users.acting_user == "ann" + + +# uses + + +SSO = """ +[site] +name = "sso" +domains = ["sso.test"] + +[source] +sdk = "sso:FakeSso" + +[data.collections.accounts] + +[data.collections.grants] +owner = "user" + +[users] +personas = [{ id = "ann", name = "Ann" }] + +[roles] +admin = ["revoke"] + +[routes.ops] +login = "POST /login" +userinfo = "GET /userinfo" +revoke = "POST /revoke" +""" + +HOST = """ +[site] +name = "intranet" +domains = ["intranet.test"] + +[source.image] +dockerfile = "Dockerfile" + +[data.collections.users] + +[users] +personas = ["ann", "ben"] + +[roles] +editor = ["sso.login"] + +[routes.ops] +pages = "GET /api/pages" +""" + + +def test_uses_mounts_a_component_prefixing_its_routes_and_namespacing_its_data(tmp_path: Path) -> None: + write(tmp_path, {"components/sso.toml": SSO}) + spec = load(write(tmp_path, {"site.toml": HOST + '[[uses]]\nspec = "components/sso.toml"\nmount = "/auth"\n' + 'data = { accounts = "users" }\n'})) + table = spec.route_table() + assert {n: (r.method, r.path, r.roles) for n, r in table.items()} == { + "pages": ("GET", "/api/pages", ()), "sso.login": ("POST", "/auth/login", ("editor",)), + "sso.userinfo": ("GET", "/auth/userinfo", ()), "sso.revoke": ("POST", "/auth/revoke", ("admin",))} + assert list(spec.data.collections) == ["users", "sso.grants"] + assert spec.data.collections["sso.grants"].owner == "user" + assert [p.id for p in spec.users.personas] == ["ann", "ben"] + assert spec.site.domains == ("intranet.test",) and spec.uses == () + (component,) = spec.components + assert (component.name, component.mount, component.spec.site.name) == ("sso", "/auth", "sso") + assert [r["name"] for r in spec.to_card_extension()["routes"]] == ["pages", "sso.login", "sso.userinfo", + "sso.revoke"] + + +def test_uses_names_a_component_by_as_and_mounts_it_at_the_root(tmp_path: Path) -> None: + write(tmp_path, {"sso.toml": SSO}) + spec = load(write(tmp_path, {"site.toml": HOST.replace("sso.login", "auth.login") + + '[[uses]]\nspec = "sso.toml"\nas = "auth"\n'})) + assert spec.route_table()["auth.login"].path == "/login" and "auth.accounts" in spec.data.collections + + +def test_a_mounted_route_keeps_its_trailing_slash(tmp_path: Path) -> None: + write(tmp_path, {"sso.toml": SSO.replace('revoke = "POST /revoke"', 'revoke = "POST /revoke"\nhome = "GET /"\n' + 'items = "GET /items/"')}) + table = load(write(tmp_path, {"site.toml": HOST + '[[uses]]\nspec = "sso.toml"\nmount = "/auth"\n'})).route_table() + assert (table["sso.home"].path, table["sso.items"].path, table["sso.login"].path) == ("/auth/", "/auth/items/", + "/auth/login") + + +def test_a_sites_acting_user_may_be_a_persona_only_its_component_declares(tmp_path: Path) -> None: + write(tmp_path, {"sso.toml": SSO}) + host = HOST.replace('personas = ["ann", "ben"]', 'personas = ["ben"]\nacting_user = "ann"') + spec = load(write(tmp_path, {"site.toml": host + '[[uses]]\nspec = "sso.toml"\nmount = "/auth"\n'})) + assert ([p.id for p in spec.users.personas], spec.users.acting_user) == (["ben", "ann"], "ann") + write(tmp_path, {"site.toml": host.replace('acting_user = "ann"', 'acting_user = "zed"') + + '[[uses]]\nspec = "sso.toml"\nmount = "/auth"\n'}) + assert problems(tmp_path / "site.toml") == [ + f"{tmp_path / 'site.toml'}: users: acting_user 'zed' is none of the personas ['ben', 'ann']"] + + +@pytest.mark.parametrize("host, component, problem", [ + (HOST + '[routes.ops.mine]\nmethod = "POST"\npath = "/auth/login"\n', SSO, + "uses[0]: route mine (POST /auth/login) is under /auth, where sso is mounted"), + (HOST.replace('pages = "GET /api/pages"', 'pages = "GET /userinfo"'), SSO.replace("[site]", "[site]\n"), + "uses[0]: sso's route userinfo is GET /userinfo, which pages already serves"), + (HOST + '[routes.ops."sso.login"]\nmethod = "GET"\npath = "/x"\n', SSO, + "uses[0]: route sso.login is declared by the site and by component sso"), + (HOST + '[data.collections.people]\nkey = "email"\n', SSO, + "uses[0]: sso's collection accounts maps onto people, whose key differ"), + (HOST, SSO.replace('[data.collections.grants]\nowner = "user"', '[data.collections.grants]\nowner = "owner"'), + "uses[0]: sso's collection grants maps onto sso.grants"), + (HOST.replace('personas = ["ann", "ben"]', 'personas = ["ann", "ben"]\nsign_in = "none"'), SSO, + "uses[0]: sso signs in by cookie, the site by none"), + (HOST.replace('personas = ["ann", "ben"]', 'personas = [{ id = "ann", name = "Annie" }]'), SSO, + "uses[0]: persona ann is declared otherwise by sso"), + (HOST.replace('editor = ["sso.login"]', 'editor = ["sso.nope"]'), SSO, "roles.editor: names no route ['sso.nope']"), + (HOST, SSO.replace('name = "sso"\n', ""), "uses[0]: components/sso.toml has no site.name; name it with as"), + (HOST, SSO.replace('login = "POST /login"', 'login = "FETCH /login"'), "routes.ops.login: method is one of"), +], ids=["host-route-under-mount", "same-path", "same-name", "collection-key", "collection-owner", "sign-in", + "persona", "role-of-component-route", "unnamed", "component-invalid"]) +def test_a_composition_that_clashes_is_refused(tmp_path: Path, host: str, component: str, problem: str) -> None: + mount = "/" if "already serves" in problem else "/auth" + mapping = 'data = { accounts = "people" }\n' if "people" in problem else "" + if "grants maps onto sso.grants" in problem: + host += '[data.collections."sso.grants"]\nowner = "user"\n' + write(tmp_path, {"components/sso.toml": component}) + found = problems(write(tmp_path, {"site.toml": host + f'[[uses]]\nspec = "components/sso.toml"\nmount = "{mount}"\n' + + mapping})) + assert any(problem in p for p in found), found + + +def test_a_components_schema_file_stays_its_own_once_mounted(tmp_path: Path) -> None: + write(tmp_path, {"components/sso.toml": SSO.replace("[data.collections.accounts]", + '[data.collections.accounts]\nschema = "accounts.json"')}) + spec = load(write(tmp_path, {"site.toml": HOST + '[[uses]]\nspec = "components/sso.toml"\n'})) + assert spec.path(spec.data.collections["sso.accounts"].schema_).resolve() == tmp_path / "components" / "accounts.json" + mapped = load(write(tmp_path, {"mapped.toml": HOST.replace("[data.collections.users]", + '[data.collections.users]\nschema = "components/accounts.json"') + + '[[uses]]\nspec = "components/sso.toml"\ndata = { accounts = "users" }\n'})) + assert mapped.data.collections["users"].schema_ == "components/accounts.json" + + +def test_a_mapping_of_a_collection_the_component_lacks_and_a_uses_cycle_are_refused(tmp_path: Path) -> None: + write(tmp_path, {"sso.toml": SSO}) + assert any("data maps ['nope'], which sso doesn't declare" in p for p in problems( + write(tmp_path, {"a.toml": HOST + '[[uses]]\nspec = "sso.toml"\ndata = { nope = "users" }\n'}))) + write(tmp_path, {"x.toml": '[site]\nname = "x"\n[[uses]]\nspec = "y.toml"\n', + "y.toml": '[site]\nname = "y"\n[[uses]]\nspec = "x.toml"\n'}) + assert any("is a cycle" in p for p in problems(tmp_path / "x.toml")) + + +def test_an_overlay_adds_a_component_and_replaces_one_of_the_same_name(tmp_path: Path) -> None: + write(tmp_path, {"sso.toml": SSO, "base.toml": HOST + '[[uses]]\nspec = "sso.toml"\nmount = "/a"\nas = "sso"\n'}) + spec = load(write(tmp_path, {"child.toml": 'extends = "base.toml"\n[[uses]]\nspec = "sso.toml"\nmount = "/b"\n' + 'as = "sso"\n[[uses]]\nspec = "sso.toml"\nmount = "/c"\nas = "backup"\n'})) + assert [(c.name, c.mount) for c in spec.components] == [("sso", "/b"), ("backup", "/c")] + assert (spec.route_table()["sso.login"].path, spec.route_table()["backup.login"].path) == ("/b/login", "/c/login") + + +def test_overlay_replaces_a_mounted_component_as_a_file_overlay_does(tmp_path: Path) -> None: + write(tmp_path, {"sso.toml": SSO, "base.toml": HOST + '[[uses]]\nspec = "sso.toml"\nmount = "/a"\nas = "sso"\n'}) + by_file = load(write(tmp_path, {"child.toml": 'extends = "base.toml"\n[[uses]]\nspec = "sso.toml"\nmount = "/b"\n' + 'as = "sso"\n'})) + by_code = load(tmp_path / "base.toml").overlay({"uses": [{"spec": "sso.toml", "mount": "/b", "as": "sso"}]}) + assert [(c.name, c.mount) for c in by_code.components] == [("sso", "/b")] + assert by_code.to_dict() == by_file.to_dict() + + +# The name table + + +def _site(name: str, domains: list[str], source: dict | None = None) -> website_spec.WebsiteSpec: + raw: dict = {"site": {"name": name, "domains": domains}} + if source is not None: + raw["source"] = source + if "live" in source or "replay" in source: + raw["users"] = {"sign_in": "live"} + return resolve(raw, Path.cwd()) + + +def test_the_name_table_puts_synthetic_before_live_before_allowlisted() -> None: + slack = load(SPECS_DIR / "slack.toml") + status = _site("status", ["status.slack.com", "*.slack-edge.com"], {"live": {"allow_hosts": ["cdn.status.io"]}}) + notes = load(NOTES_DIR / "website.toml") + table = name_table([slack, status, notes]) + assert table.as_dict() == { + "slack.com": {"site": "slack", "kind": "synthetic"}, "*.slack.com": {"site": "slack", "kind": "synthetic"}, + "*.slack-edge.com": {"site": "status", "kind": "live"}, "status.slack.com": {"site": "status", "kind": "live"}, + "cdn.status.io": {"site": "status", "kind": "allowlist"}, "notes.test": {"site": "notes", "kind": "synthetic"}} + assert [(lost.site, lost.kind, won.site) for lost, won in table.shadowed] == [("slack", "allowlist", "status")] + assert table.resolve("status.slack.com").site == "slack" + assert table.resolve("files.slack-edge.com").kind == "live" + assert table.resolve("NOTES.test.").site == "notes" and table.resolve("slack.org") is None + + +def test_the_name_table_prefers_the_most_specific_pattern_of_one_kind() -> None: + table = name_table([_site("mail", ["*.example.test"]), _site("inbox", ["inbox.example.test"]), + _site("deep", ["*.b.example.test"])]) + assert [table.resolve(h).site for h in ("x.example.test", "inbox.example.test", "a.b.example.test")] == [ + "mail", "inbox", "deep"] + assert table.resolve("example.test") is None + + +def test_the_name_table_refuses_two_sites_claiming_one_name_at_one_kind() -> None: + with pytest.raises(SpecError) as caught: + name_table([_site("a", ["shop.test", "*.x.test"]), _site("b", ["shop.test"]), _site("c", ["*.x.test"]), + _site("a", []), resolve({"site": {"domains": ["q.test"]}}, Path.cwd()), + _site("l1", ["live.test"], {"live": {}}), _site("l2", ["live.test"], {"live": {}})]) + assert caught.value.problems == [ + "two sites are named a", "a site with domains ['q.test'] has no site.name", + "shop.test: claimed as synthetic by a, b", "*.x.test: claimed as synthetic by a, c", + "live.test: claimed as live by l1, l2"] + + + +def test_the_name_table_refuses_two_live_sites_on_one_name_a_synthetic_site_also_claims() -> None: + with pytest.raises(SpecError) as caught: + name_table([_site("l1", ["shop.test"], {"live": {}}), _site("l2", ["shop.test"], {"live": {}}), + _site("s", ["shop.test"])]) + assert caught.value.problems == ["shop.test: claimed as live by l1, l2"] + + +def test_two_sites_may_allowlist_one_host() -> None: + table = name_table([_site("a", [], {"live": {"allow_hosts": ["cdn.test"]}}), + _site("b", [], {"replay": {"capture": "c.har", "allow_hosts": ["cdn.test"]}})]) + assert table.as_dict() == {"cdn.test": {"site": "a", "kind": "allowlist"}} + + +# The card + + +def test_a_spec_round_trips_through_the_card_it_advertises() -> None: + spec = load(NOTES_DIR / "website.toml") + card = {"name": "notes", "capabilities": {"extensions": [ + {"uri": website_spec.WEBSITE_EXTENSION_URI, "params": spec.to_card_extension()}]}} + back = from_card(card) + assert back.to_card_extension() == spec.to_card_extension() + assert (back.site, back.users.sign_in, back.clock) == (spec.site, "cookie", spec.clock) + assert back.source is None and back.data.collections == {} + + +def test_from_card_reads_a_running_sites_card_and_a_card_without_a_session() -> None: + card = {"name": "docs", "capabilities": {"extensions": [{"uri": website_spec.WEBSITE_EXTENSION_URI, "params": { + "domains": ["docs.test"], "health": "/up", "routes": [{"method": "GET", "path": "/a", "name": "a"}]}}]}} + spec = from_card(card) + assert (spec.site.name, spec.health, spec.users.sign_in, spec.clock) == ("docs", "/up", "none", None) + assert spec.to_card_extension() == card["capabilities"]["extensions"][0]["params"] | { + "routes": [{"method": "GET", "path": "/a", "name": "a", "roles": None}]} + with pytest.raises(SpecError, match="declares no urn:agentenv:website/v1"): + from_card({"name": "x"}) + + +def test_the_notes_sites_card_is_its_spec(notes_client: TestClient) -> None: + from agentenv_protocol.website.routes import openapi_path + + card = card_of(notes_client) + params = website_params(card) + assert card["name"] == "notes" + assert {r["name"]: r["path"] for r in params["routes"]}["get_note"] == "/api/notes/{note_id:int}" + templated = [{**r, "path": openapi_path(r["path"])} for r in params["routes"]] + assert {**params, "routes": templated} == load(NOTES_DIR / "website.toml").to_card_extension() + assert from_card(card).to_card_extension() == {**params, "routes": templated} + + +# WebsiteEnvironment + + +def test_from_environment_derives_the_notes_spec_from_its_class() -> None: + derived = from_environment(notes_site.Notes) + declared = load(NOTES_DIR / "website.toml") + assert derived.to_card_extension() == declared.to_card_extension() + assert (derived.source.kind, derived.source.sdk) == ("sdk", f"{notes_site.__name__}:Notes") + assert derived.users.acting_user == "ann" and derived.path(derived.data.seed[0]) == NOTES_DIR / "data" / "seed.json" + + +def test_from_environment_derives_a_spec_from_decorators_alone(tmp_path: Path) -> None: + (tmp_path / "seed.json").write_text('{"current_user": "zoe", "items": []}') + + @environment_card(name="shop") + @website(domains=["shop.test"], health="/up", data=str(tmp_path / "seed.json"), clock_patch=True) + class Shop(WebsiteEnvironment): + @api.get("/api/items", roles=["admin"]) + def items(self) -> list: + return [] + + spec = from_environment(Shop) + assert (spec.site.name, spec.site.domains, spec.health, spec.users.acting_user) == ("shop", ("shop.test",), "/up", + "zoe") + assert (spec.clock.follow, spec.route_table()["items"].roles) == ("date_patch", ("admin",)) + assert website_params(card_of(TestClient(Shop().create_app()))) == spec.to_card_extension() + + +def test_from_environment_takes_any_url_path_a_route_serves() -> None: + @environment_card(name="chat") + @website(domains=["chat.test"]) + class Chat(WebsiteEnvironment): + @api.get("/api/users/@me") + def me(self) -> dict: + return {"id": "me"} + + @api.get("/api/tags/c++,%2F") + def tags(self) -> list: + return [] + + assert {r.path for r in from_environment(Chat).route_table().values()} == {"/api/users/@me", "/api/tags/c++,%2F"} + assert TestClient(Chat().create_app()).get("/api/users/@me").json() == {"id": "me"} + + +def test_from_environment_keeps_any_name_an_api_route_takes() -> None: + @environment_card(name="shop") + @website(domains=["shop.test"]) + class Shop(WebsiteEnvironment): + @api.get("/api/items", name="items:list") + def items(self) -> list: + return [] + + assert list(from_environment(Shop).route_table()) == ["items:list"] + card = card_of(TestClient(Shop().create_app())) + assert [r["name"] for r in website_params(card)["routes"]] == ["items:list"] + assert list(from_card(card).route_table()) == ["items:list"] + for derived in (from_environment(Shop), from_card(card)): + assert derived.overlay({"site": {"domains": ["shop.example"]}}).site.domains == ("shop.example",) + assert "routes.ops: route names ['items:list'] aren't names" in problems( + {"routes": {"ops": {"items:list": "GET /api/items"}}}, Path.cwd())[0] + + +def test_a_site_whose_domains_are_in_upper_case_starts_and_its_spec_names_them_in_lower_case() -> None: + @environment_card(name="notes") + @website(domains=["Notes.TEST", "*.Notes.Test"]) + class Notes(WebsiteEnvironment): + pass + + card = card_of(TestClient(Notes().create_app())) + assert from_environment(Notes).site.domains == ("notes.test", "*.notes.test") + assert from_card(card).site.domains == ("notes.test", "*.notes.test") + + +def test_from_environment_keeps_a_head_or_options_route_an_api_route_takes() -> None: + @environment_card(name="shop") + @website(domains=["shop.test"]) + class Shop(WebsiteEnvironment): + @api.route("HEAD", "/ping") + def ping(self) -> dict: + return {} + + @api.route("OPTIONS", "/items") + def items_options(self) -> dict: + return {} + + assert {name: (r.method, r.path) for name, r in from_environment(Shop).route_table().items()} == { + "ping": ("HEAD", "/ping"), "items_options": ("OPTIONS", "/items")} + routes = website_params(card_of(TestClient(Shop().create_app())))["routes"] + assert [(r["method"], r["name"]) for r in routes] == [("HEAD", "ping"), ("OPTIONS", "items_options")] + + +def test_a_class_declared_from_a_spec_must_serve_what_it_declares(tmp_path: Path) -> None: + path = write(tmp_path, {"site.toml": '[site]\nname = "wiki"\ndomains = ["wiki.test"]\n[source]\nsdk = "m:Wiki"\n' + '[routes.ops]\npages = "GET /api/pages"\nedit = "PUT /api/pages/{id}"\n' + '[roles]\nadmin = ["edit"]\n'}) + + class Wiki(WebsiteEnvironment): + @api.get("/api/pages") + def pages(self) -> list: + return [] + + @api.put("/api/pages/{id}", name="edit") + def edit_page(self, id: str) -> dict: + return {} + + @api.get("/api/extra") + def extra(self) -> dict: + return {} + + with pytest.raises(SpecError) as caught: + website(spec=path)(Wiki) + assert [p.split(": ", 1)[1] for p in caught.value.problems] == [ + "routes.ops.edit: the spec says PUT /api/pages/{id} for ['admin'], the class serves PUT /api/pages/{id}", + "routes.ops: Wiki serves extra (GET /api/extra), which the spec doesn't declare"] + with pytest.raises(SpecError, match="source.sdk: m:Wiki names another class than Other"): + website(spec=load(path).overlay({"routes": {"ops": {"edit": False}}, "roles": {"admin": False}}))( + type("Other", (WebsiteEnvironment,), {})) + with pytest.raises(TypeError, match="takes the site from its spec"): + website(spec=path, domains=["x.test"]) + + +def test_a_class_declared_from_a_spec_is_named_and_seeded_by_it(tmp_path: Path) -> None: + write(tmp_path, {"a.json": '{"current_user": "ann", "pages": [{"id": 1}]}', "b.json": '{"pages": [{"id": 2}]}', + "dist/index.html": ""}) + path = write(tmp_path, {"site.toml": '[site]\nname = "wiki"\ndomains = ["wiki.test"]\nhealth = "/up"\n' + '[source]\nsdk = "m:Wiki"\nassets = "dist"\n[data]\nseed = ["a.json", "b.json"]\n' + '[clock]\nfollow = "date_patch"\n'}) + + @website(spec=path) + class Wiki(WebsiteEnvironment): + pass + + env = Wiki() + client = TestClient(env.create_app()) + assert card_of(client)["name"] == "wiki" and client.get("/up").status_code == 200 + assert env.store["pages"] == [{"id": 1}, {"id": 2}] and env.store.current_user == "ann" + assert Wiki.declaration().clock_patch and " + + +
    +

    {{title}}

    +
    + + +
    +
    +
    + +
    +
    + + + diff --git a/src/agent_env/cli/env/website_template/web/package.json.tmpl b/src/agent_env/cli/env/website_template/web/package.json.tmpl new file mode 100644 index 000000000..a4af53d1d --- /dev/null +++ b/src/agent_env/cli/env/website_template/web/package.json.tmpl @@ -0,0 +1,14 @@ +{ + "name": "{{name}}-site-web", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "dev": "vite", + "build": "vite build", + "preview": "vite preview" + }, + "devDependencies": { + "vite": "5.4.10" + } +} diff --git a/src/agent_env/cli/env/website_template/web/src/main-tools.js.tmpl b/src/agent_env/cli/env/website_template/web/src/main-tools.js.tmpl new file mode 100644 index 000000000..d0e5926fd --- /dev/null +++ b/src/agent_env/cli/env/website_template/web/src/main-tools.js.tmpl @@ -0,0 +1,68 @@ +import "./style.css"; +import { api, followUniverseClock, session, signIn, universeNow } from "./site-client.js"; + +// The routes `agent-env env website new` made from the MCP server's tools. Each answers 501 until site.py writes it. +const ROUTES = {{routes_json}}; + +const $ = (id) => document.getElementById(id); + +function showError(error) { + $("error").textContent = error ? error.error || error.message : ""; + $("error").hidden = !error; +} + +function tick() { + const now = universeNow(); + $("universe-now").textContent = `${now.toLocaleString(undefined, { dateStyle: "medium", timeStyle: "short", timeZone: "UTC" })} UTC`; + $("universe-now").dateTime = now.toISOString(); +} + +function routeItem(route) { + const item = document.createElement("li"); + item.className = "card"; + const title = document.createElement("code"); + title.textContent = `${route.method} ${route.path}`; + const about = document.createElement("p"); + about.textContent = route.description || route.name; + const params = document.createElement("textarea"); + params.rows = 2; + params.value = JSON.stringify(route.example, null, 0); + params.setAttribute("aria-label", `${route.name} parameters`); + const call = document.createElement("button"); + call.textContent = "Call"; + const out = document.createElement("pre"); + call.addEventListener("click", async () => { + try { + const values = JSON.parse(params.value || "{}"); + const query = new URLSearchParams(Object.entries(values).map(([k, v]) => [k, String(v)])).toString(); + const answer = route.method === "GET" + ? await api(query ? `${route.path}?${query}` : route.path) + : await api(route.path, { method: route.method, body: values }); + out.textContent = JSON.stringify(answer, null, 2); + showError(null); + } catch (error) { + out.textContent = ""; + showError(error); + } + }); + item.append(title, about, params, call, out); + return item; +} + +async function start() { + const [signedIn, users] = await Promise.all([session(), api("api/users")]); + $("user").replaceChildren(...users.map((user) => new Option(user.name || user.id, user.id))); + $("user").value = signedIn.user ?? ""; + $("user").addEventListener("change", (event) => signIn(event.target.value).then(() => showError(null), showError)); + + const list = document.createElement("ul"); + list.id = "routes"; + list.append(...ROUTES.map(routeItem)); + $("app").append(list); + + tick(); + setInterval(tick, 1000); + followUniverseClock({ path: "healthz" }); +} + +start().catch(showError); diff --git a/src/agent_env/cli/env/website_template/web/src/main.js.tmpl b/src/agent_env/cli/env/website_template/web/src/main.js.tmpl new file mode 100644 index 000000000..06c34f466 --- /dev/null +++ b/src/agent_env/cli/env/website_template/web/src/main.js.tmpl @@ -0,0 +1,85 @@ +import "./style.css"; +import { api, followUniverseClock, session, signIn, universeNow } from "./site-client.js"; + +const $ = (id) => document.getElementById(id); + +function showError(error) { + $("error").textContent = error ? error.error || error.message : ""; + $("error").hidden = !error; +} + +function formatTime(value) { + return new Date(value).toLocaleString(undefined, { dateStyle: "medium", timeStyle: "short", timeZone: "UTC" }); +} + +function tick() { + $("universe-now").textContent = `${formatTime(universeNow())} UTC`; + $("universe-now").dateTime = universeNow().toISOString(); +} + +async function render() { + const items = await api("api/items"); + const list = document.createElement("ul"); + list.id = "items"; + for (const item of items) { + const row = document.createElement("li"); + row.className = "card"; + row.textContent = item.title; + const meta = document.createElement("small"); + meta.textContent = ` · ${formatTime(item.created_at)}`; + row.append(meta); + list.append(row); + } + if (!items.length) { + const empty = document.createElement("li"); + empty.className = "empty"; + empty.textContent = "Nothing here yet."; + list.append(empty); + } + $("items")?.remove(); + $("app").append(list); +} + +async function run(action) { + try { + await action(); + showError(null); + await render(); + } catch (error) { + showError(error); + } +} + +async function start() { + const [signedIn, users] = await Promise.all([session(), api("api/users")]); + $("user").replaceChildren(...users.map((user) => new Option(user.name || user.id, user.id))); + $("user").value = signedIn.user ?? ""; + $("user").addEventListener("change", (event) => run(() => signIn(event.target.value))); + + const form = document.createElement("form"); + form.id = "new-item"; + form.className = "card"; + const title = document.createElement("input"); + title.placeholder = "New item"; + title.required = true; + title.setAttribute("aria-label", "New item"); + const add = document.createElement("button"); + add.type = "submit"; + add.textContent = "Add"; + form.append(title, add); + form.addEventListener("submit", (event) => { + event.preventDefault(); + run(async () => { + await api("api/items", { method: "POST", body: { title: title.value } }); + form.reset(); + }); + }); + $("app").append(form); + + tick(); + setInterval(tick, 1000); + followUniverseClock({ path: "healthz" }); + await render(); +} + +start().catch(showError); diff --git a/src/agent_env/cli/env/website_template/web/src/style.css.tmpl b/src/agent_env/cli/env/website_template/web/src/style.css.tmpl new file mode 100644 index 000000000..ba2c7a4fa --- /dev/null +++ b/src/agent_env/cli/env/website_template/web/src/style.css.tmpl @@ -0,0 +1,46 @@ +:root { + --bg: #f6f6f3; + --card: #ffffff; + --text: #1d1d1b; + --muted: #6b6b66; + --line: #deded8; + --accent: #2f5bd3; + font-family: system-ui, -apple-system, "Segoe UI", sans-serif; + color: var(--text); + background: var(--bg); +} + +* { box-sizing: border-box; } +body { margin: 0; } + +.bar { + display: flex; + flex-wrap: wrap; + gap: 12px; + align-items: center; + justify-content: space-between; + padding: 12px 16px; + border-bottom: 1px solid var(--line); + background: var(--card); +} +.bar h1 { margin: 0; font-size: 1.25rem; } +.who { display: flex; flex-wrap: wrap; gap: 12px; align-items: center; color: var(--muted); font-size: 0.9rem; } +.who time { font-variant-numeric: tabular-nums; } + +main { max-width: 720px; margin: 0 auto; padding: 16px; } + +.card { background: var(--card); border: 1px solid var(--line); border-radius: 8px; padding: 12px; } + +input, textarea, select, button { font: inherit; color: inherit; } +input, textarea { width: 100%; border: 1px solid var(--line); border-radius: 6px; padding: 6px 8px; background: transparent; } +button { border: 1px solid var(--line); border-radius: 6px; padding: 6px 12px; background: var(--card); cursor: pointer; } +button[type="submit"] { background: var(--accent); border-color: var(--accent); color: #fff; } + +form.card { display: flex; gap: 8px; margin-bottom: 16px; } +ul { list-style: none; margin: 0; padding: 0; display: grid; gap: 8px; } +li small { color: var(--muted); } +li pre { margin: 8px 0 0; white-space: pre-wrap; font-size: 0.85rem; } +li textarea { margin: 8px 0; font-family: ui-monospace, monospace; font-size: 0.85rem; } +.empty { color: var(--muted); text-align: center; padding: 24px; } + +#error { color: #b3261e; margin: 0 0 12px; } diff --git a/src/agent_env/cli/env/website_template/web/vite.config.js.tmpl b/src/agent_env/cli/env/website_template/web/vite.config.js.tmpl new file mode 100644 index 000000000..79f723c86 --- /dev/null +++ b/src/agent_env/cli/env/website_template/web/vite.config.js.tmpl @@ -0,0 +1,11 @@ +import { defineConfig } from "vite"; + +export default defineConfig({ + base: "./", + server: { + proxy: { + "/api": process.env.SITE_URL || "http://127.0.0.1:8000", + "/healthz": process.env.SITE_URL || "http://127.0.0.1:8000", + }, + }, +}); diff --git a/src/agent_env/env/env.py b/src/agent_env/env/env.py index 411cceb91..6dd6e2b19 100644 --- a/src/agent_env/env/env.py +++ b/src/agent_env/env/env.py @@ -9,6 +9,7 @@ from typing import TYPE_CHECKING, Any, ClassVar, Optional, Self from agentenv_protocol import client as protocol_v1 +from agentenv_protocol.website import READY_TIMEOUT_S, WEBSITE_EXTENSION_URI, WEBSITE_READY_OP, WebsiteReadiness, env_readiness from agent_env.entity.refs import EntityRef from agent_env.plugins._bundles import ENV_ALIASES, installed_aliases @@ -16,7 +17,7 @@ from agent_env.store.routing import namespace_routing_enabled from agent_env.utils.paths import validate_relative_filename -from .gateway import GatewayMode +from .gateway import AGENT_ENV_ROLE_HEADER, AGENT_ENV_ROLE_KEY_HEADER, GatewayMode from .gateway.constants import WELL_KNOWN_PATH from .store import get_env_store @@ -82,6 +83,7 @@ class DeployedEnv: # The provider that made the record, so the class it loads as (see from_dict); capabilities still come from the card. env_provider_type: Optional[str] = None environment_card_url: Optional[str] = None + role_key: Optional[str] = None # The env card as last read, and when (ISO 8601, UTC); written at deploy from the readiness probe. environment_card: Optional[dict] = None environment_card_read_at_utc: Optional[str] = None @@ -121,6 +123,11 @@ def control_headers_for(self, url: str) -> dict[str, str]: """What a call to ``url`` sends of the env's control token: ``control_headers`` when ``url`` is the env's address or under it, as a step's grant gives them, and none anywhere else, an override's address say.""" return protocol_v1.granted_headers(url, {self.environment_url: self.control_headers}) + def role_headers(self, role: str) -> dict[str, str]: + """The headers a request to the env sends to act as `role`: the role, and the deploy's `role_key` beside it, + without which a gateway that has one takes the request for `default`'s. A record from before the key, of a deploy without + `keyed_roles`, or of one with no gateway, has none, and its role header alone is believed.""" + return {AGENT_ENV_ROLE_HEADER: role, **({AGENT_ENV_ROLE_KEY_HEADER: self.role_key} if self.role_key else {})} def supports(self, uri: str, method: str) -> bool: """Whether the stored env card offers `method` on the extension at `uri`; no network.""" @@ -130,26 +137,59 @@ def require(self, uri: str, method: str) -> None: if not self.supports(uri, method): raise EnvCapabilityUnsupported(uri, method, self.env_id, self.instance_id) - def get_child_env_card(self, environment_name: str) -> Optional[dict]: + def get_child_env_card(self, environment_name: str, *, mcp: Optional[bool] = None) -> Optional[dict]: + """The card of the stored card's child env named `environment_name` (``child_env``), or None.""" + child = self.child_env(environment_name, mcp=mcp) + return child.card if child is not None else None + + def child_env(self, environment_name: str, *, mcp: Optional[bool] = None) -> Optional[protocol_v1.ChildEnv]: """The stored card's child env named `environment_name`; a leaf card is its own only child env, and a composed card serving a data plane of its own, as a computer's driver fronting its apps does, is the one its own name - reaches.""" + reaches. Of several children of a composed card sharing the name, with `mcp`, only an MCP server's (True) or a + website's (False, ``website_child``) is, so a website and an MCP server sharing a name are told apart. Raises + ``AmbiguousChildError`` when several children still are: their urls tell them apart (``protocol_v1.child_at``).""" card = self.environment_card or {} composed = isinstance(card.get("children_environments"), list) - own = not composed or protocol_v1.operations(card) - if own and card.get("name") == environment_name: - return card - return protocol_v1.find_child(card, environment_name) if composed else None + if (not composed or protocol_v1.operations(card)) and card.get("name") == environment_name: + return protocol_v1.ChildEnv(name=card.get("name"), url=card.get("url"), card=card) + found = [child for child in protocol_v1.children(card) if child.name == environment_name] + if len(found) > 1 and mcp is not None: + found = [child for child in found if website_child(child) is not mcp] + if len(found) > 1: + raise protocol_v1.AmbiguousChildError( + f"{len(found)} child envs of env '{self.env_id}' are named {environment_name!r} " + f"({', '.join(str(child.url) for child in found)}); look one up by its url") + return found[0] if found else None async def invoke(self, uri: str, method: str, params: Optional[dict] = None, *, - environment_name: Optional[str] = None, timeout: int = _EXTENSION_CALL_TIMEOUT_S) -> Any: - """Call `method` on the extension at `uri` as the env card, or a child env's card, advertises it.""" - card = self.environment_card if environment_name is None else self.get_child_env_card(environment_name) + environment_name: Optional[str] = None, child: Optional[protocol_v1.ChildEnv] = None, + timeout: int = _EXTENSION_CALL_TIMEOUT_S) -> Any: + """Call `method` on the extension at `uri` as the env card advertises it, or as a child env's card does: the + `child` given, one of the card's (``protocol_v1.children``), else the one `environment_name` names + (``child_env``).""" + if child is not None: + card = child.card + environment_name = child.name + else: + card = self.environment_card if environment_name is None else self.get_child_env_card(environment_name) if card is None or not self.environment_url or protocol_v1.find_extension_method(card, uri, method) is None: raise EnvCapabilityUnsupported(uri, method, self.env_id, self.instance_id, environment_name) return await protocol_v1.invoke_extension(self.environment_url, card, uri, params, timeout, method=method, headers=self.control_headers) + async def readiness(self, timeout: float = READY_TIMEOUT_S) -> WebsiteReadiness: + """Whether the env is ready, as the protocol determines a website env's health (``env_readiness``): every site + its card, the stored one or, for a record without it, the card it serves now, advertises ``website/v1``'s + ``ready`` on is asked, with the env's control headers, each within ``timeout``, and its problems are named + after it. An env with no site serving ``ready``, as one of MCP servers alone or a computer, is ready with no + problems. Raises ``EnvCapabilityUnsupported`` for a record with no card URL, which nothing can be asked at.""" + if not self.environment_url: + raise EnvCapabilityUnsupported(WEBSITE_EXTENSION_URI, WEBSITE_READY_OP, self.env_id, self.instance_id) + card = self.environment_card + if card is None: + card = await protocol_v1.get_card(self.environment_url, timeout=timeout) + return await env_readiness(self.environment_url, card, timeout=timeout, headers=self.control_headers) + @classmethod def _fields_from(cls, data: dict) -> dict: return dict( @@ -166,6 +206,7 @@ def _fields_from(cls, data: dict) -> dict: created_at_utc=data.get("created_at_utc"), expires_at_utc=data.get("expires_at_utc"), control_token=data.get("control_token"), + role_key=data.get("role_key"), ) @@ -380,6 +421,11 @@ def deploy_refusal(self, *, sandbox_type: Optional[str] = None, env_params: Opti overrides this; one that doesn't takes no ``env_params``.""" return f"{self.type} envs take no env_params" if env_params else None + async def run_env_params(self, env_params: Optional[dict], deployed_envs: list[DeployedEnv]) -> Optional[dict]: + """``env_params`` as a run's deploy_env passes them to deploy(), once the run has deployed ``deployed_envs``: + as given, unless the type reads something of what the run deployed into them.""" + return env_params + @classmethod async def from_deployed_env(cls, deployed: DeployedEnv) -> Self: """The env a running deployment serves, for its caller to drive. A type that can be @@ -466,6 +512,13 @@ async def load_file_artifact_universe( return LoadFileArtifactUniverseResult(destination_path=destination, files=loaded) +def website_child(child: protocol_v1.ChildEnv) -> bool: + """Whether a child env is a website: its card carries ``website/v1``, or, for a site serving none, its url is at a + website's gateway route, ``/svc/{name}`` (an MCP server's is ``/svc/mcp-{name}``).""" + route, url = f"/svc/{child.name}", (child.url or "").rstrip("/") + return child.find_extension(WEBSITE_EXTENSION_URI) is not None or url == route or url.startswith(f"{route}/") + + def gateway_url_of(deployed: Optional[DeployedEnv]) -> Optional[str]: """The gateway's URL, or None for a record without a gateway.""" return deployed.gateway_url if isinstance(deployed, DeployedGatewayEnv) else None diff --git a/src/agent_env/env/envs/_deployment.py b/src/agent_env/env/envs/_deployment.py index fc097d55b..6dc06a735 100644 --- a/src/agent_env/env/envs/_deployment.py +++ b/src/agent_env/env/envs/_deployment.py @@ -209,8 +209,9 @@ def check_plugin_record(env: Env, deployed: DeployedEnv) -> None: "of its MCP servers and websites") return if isinstance(env, MultiEnv): - names = [child.environment_name for child in [*env.mcp_server_envs, *env.website_envs]] - if missing := [name for name in names if deployed.get_child_env_card(name) is None]: + names = [(child.environment_name, True) for child in env.mcp_server_envs] + names += [(child.environment_name, False) for child in env.website_envs] + if missing := [name for name, mcp in names if deployed.get_child_env_card(name, mcp=mcp) is None]: raise RuntimeError(f"{returned} whose env card lists no child env named {', '.join(map(repr, missing))}; a MultiEnv's card " "lists each of its MCP servers and websites by environment_name") if env.name and deployed.mcp_server_name != env.name: @@ -220,8 +221,9 @@ def check_plugin_record(env: Env, deployed: DeployedEnv) -> None: raise RuntimeError(f"{returned} whose env card is named {deployed.environment_card.get('name')!r}; it must be the env's " f"environment_name {env.environment_name!r}, or list a child env of that name") else: - names = [env.environment_name] - if not_paths := [name for name in names if not deployed.get_child_env_card(name).get("url", RPC_PATH).startswith("/")]: + names = [(env.environment_name, None)] + if not_paths := [name for name, mcp in names + if not deployed.get_child_env_card(name, mcp=mcp).get("url", RPC_PATH).startswith("/")]: raise RuntimeError(f"{returned} whose env card gives {', '.join(map(repr, not_paths))} a url that isn't a path; a child env's " "url is a path under the record's address, such as '/svc//agentenv'") @@ -286,13 +288,19 @@ def provider_or_class(env: Env) -> EnvironmentProvider | type[EnvironmentProvide def deploy_refusal(env: Env, provider: EnvironmentProvider | type[EnvironmentProvider], options: dict) -> str | None: """Why deploying env through provider, or a provider of that class, is refused before anything is built, or None: the provider - can't deploy an env like it (see ``provider_refusal``), or an option it can't take.""" + can't deploy an env like it (see ``provider_refusal``), ``capture_traffic`` for an env that serves no website, or an option it + can't take.""" + from agent_env.env.envs.website import WebsiteEnv + provider_class = provider if isinstance(provider, type) else type(provider) reason = provider_refusal(provider_class, type(env), mcp_names=[child.environment_name for child in getattr(env, "mcp_server_envs", ())], website_names=[child.environment_name for child in getattr(env, "website_envs", ())]) if reason is not None: return f"env '{env.id}' has env_provider_type '{env.env_provider_type}', which {reason}" + if options.get("capture_traffic") and not (isinstance(env, WebsiteEnv) or getattr(env, "website_envs", ())): + return (f"env '{env.id}' serves no website, and capture_traffic records what a website's containers send and " + "receive") return option_refusal(env, provider, options) diff --git a/src/agent_env/env/envs/computer/env.py b/src/agent_env/env/envs/computer/env.py index 5e36d9a8f..8b65d64cc 100644 --- a/src/agent_env/env/envs/computer/env.py +++ b/src/agent_env/env/envs/computer/env.py @@ -27,7 +27,7 @@ deploy_through_provider, provider_or_class, ) -from agent_env.env.envs.computer.utils import boot, composer, gate, inside +from agent_env.env.envs.computer.utils import boot, composer, gate, inside, sites from agent_env.env.envs.computer.utils.composer import Composing from agent_env.env.envs.computer.utils import params as computer_params from agent_env.env.envs.computer.driver_env import ComputerDriverEnv @@ -248,6 +248,22 @@ async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, env_state_type=env_state_type, env_state_instance_id=env_state_instance_id, attribution=attribution, env_params=env_params) + async def run_env_params(self, env_params: Optional[dict], deployed_envs: list[DeployedEnv]) -> Optional[dict]: + """With ``site_routing`` and no ``sites``, ``env_params`` naming as its sites the websites the run deployed + before the computer: all of them, or those of the env ids ``site_routing`` lists.""" + routing = (env_params or {}).get("site_routing") + if routing is not None and routing is not False and routing is not True and not ( + isinstance(routing, list) and all(isinstance(env_id, str) for env_id in routing)): + raise ValueError(f"computer env {self.id!r}: env_params.site_routing is true or a list of env ids, not " + f"{routing!r}") + if not routing or env_params.get("sites"): + return env_params + found = await sites.deployed_sites(deployed_envs, None if routing is True else list(routing)) + if not found: + raise ValueError(f"computer env {self.id!r}: env_params.site_routing reaches the run's websites, and the " + "run deployed none before the computer") + return {**env_params, "sites": found} + def deploy_refusal(self, *, sandbox_type: str | None = None, env_params: Optional[dict] = None, **options) -> str | None: """Why deploy() would refuse before booting anything, or None: an option the computer provider doesn't take, a diff --git a/src/agent_env/env/envs/computer/utils/params.py b/src/agent_env/env/envs/computer/utils/params.py index c675a6364..7d9c8feff 100644 --- a/src/agent_env/env/envs/computer/utils/params.py +++ b/src/agent_env/env/envs/computer/utils/params.py @@ -1,22 +1,37 @@ """What a deploy of a computer env takes as ``deploy_env`` ``env_params``: the form to boot, ``computer_settings``, values of the base's deploy-phase settings, ``driver_settings``, the driver's ``ComputerSettings``, such as the default -coordinate space its tools describe, and ``app_settings``, values of each app's ``Settings`` by its name. The env's -display and network are its own, which no deploy overrides.""" +coordinate space its tools describe, ``app_settings``, values of each app's ``Settings`` by its name, and +``site_routing``, opt-in, to reach the run's websites by their names from the computer: its ``sites``, which deploy_env +fills in from the websites the run deployed before the computer, all of them or those of the env ids it lists. The +env's display and network are its own, which no deploy overrides.""" from __future__ import annotations -from typing import TYPE_CHECKING, Any, Optional +from typing import TYPE_CHECKING, Any, Optional, Union from agentenv_protocol.computer import ComputerSettings -from pydantic import BaseModel, ConfigDict, Field, ValidationError +from pydantic import BaseModel, ConfigDict, Field, StrictBool, ValidationError from agent_env.artifact.artifacts.computer import SettingValue -from agent_env.env.envs.computer.utils import gate +from agent_env.env.envs.computer.utils import gate, sites if TYPE_CHECKING: from agent_env.env.envs.computer.env import ComputerEnv +class Site(BaseModel): + """A website the computer reaches by its names: its route on the env gateway, as this process reaches it, and what + its card's website/v1 declares: its names, and for a live site, which the real site serves, the hosts it calls.""" + + model_config = ConfigDict(extra="forbid") + + name: str + url: str + domains: list[str] = Field(default_factory=list) + live: bool = False + allow_hosts: list[str] = Field(default_factory=list) + + class ComputerParams(BaseModel): model_config = ConfigDict(extra="forbid") @@ -24,6 +39,8 @@ class ComputerParams(BaseModel): computer_settings: dict[str, Any] = Field(default_factory=dict) driver_settings: dict[str, Any] = Field(default_factory=dict) app_settings: dict[str, dict[str, Any]] = Field(default_factory=dict) + site_routing: Union[StrictBool, list[str]] = False + sites: list[Site] = Field(default_factory=list) def checked(env: ComputerEnv, env_params: Optional[dict]) -> tuple[ComputerParams, dict[str, SettingValue], list[str]]: @@ -55,6 +72,10 @@ def checked(env: ComputerEnv, env_params: Optional[dict]) -> tuple[ComputerParam merged = app_settings(env, params.app_settings) for name, app in apps.items(): problems += gate.check_app_settings(app, merged[name], f"env_params.app_settings.{name}", env.computer.os.type) + if params.sites and not params.site_routing: + problems.append("env_params.sites: a computer reaches sites only with site_routing") + allow_hosts = (env.network or {}).get("allow_hosts") or () + problems += sites.table_problems(params.sites, allow_hosts) if params.sites else [] return params, values, problems diff --git a/src/agent_env/env/envs/computer/utils/site_proxy.py b/src/agent_env/env/envs/computer/utils/site_proxy.py new file mode 100644 index 000000000..83ea8e9b1 --- /dev/null +++ b/src/agent_env/env/envs/computer/utils/site_proxy.py @@ -0,0 +1,903 @@ +"""The site proxy: inside a computer, it serves each website of the deployment under the names the site declares +(``https://notes.test/``), forwarding every request to the site's route on the env gateway +(``/website/notes/``), the only address the computer reaches the site at. It runs on the base's own +``python3``, so it uses the standard library alone. + +``install``, as root, maps each exact name to loopback in the hosts file, trusts the deployment's CA in the system +store, Firefox's policies, Chromium's managed policies and the session user's NSS databases, starts ``serve`` and checks, as the session user, that +each name loads. ``serve`` listens on loopback: on 443 for a name served over HTTPS, on 80 to redirect those to HTTPS +and to serve ``.localhost`` names over HTTP, and on 53 for DNS, over UDP and TCP, when a name is a wildcard, which a +hosts file can't list; it then drops to ``SERVE_USER`` (``nobody``), else the session user, so the agent, which runs as +the session user, can't stop it, and ``keep``, as root, starts it again should it end. The site sees itself at ``/`` under its own name: the path goes upstream under +the route's prefix, ``X-Forwarded-Host`` and ``X-Forwarded-Proto`` say the name and scheme the browser used, +``AgentEnv-Site-Root`` has the gateway tell the site no prefix, so the links it makes are rooted at ``/`` too, and a +``Location`` or cookie naming the route is rewritten back to the name. Hop-by-hop headers and the browser's own +``X-Forwarded-*`` never go upstream; ``Transfer-Encoding`` does, since bodies are relayed as framed, and an +``Alt-Svc`` offering HTTP/3, which the proxy doesn't serve, never reaches the browser. A WebSocket's upgrade is +relayed, for an upstream that answers it.""" + +from __future__ import annotations + +import asyncio +import json +import os +import pwd +import shutil +import socket +import ssl +import struct +import subprocess +import sys +import time +import urllib.error +import urllib.request +from dataclasses import dataclass, field +from pathlib import Path +from typing import Optional +from urllib.parse import urlsplit, urlunsplit + +HOSTS_BEGIN = "# agentenv sites" +HOSTS_END = "# end agentenv sites" +LOOPBACK = "127.0.0.1" +CA_NAME = "agentenv-sites" +NSS_NICKNAME = "agentenv sites" +HOP_BY_HOP = frozenset({"connection", "keep-alive", "proxy-connection", "proxy-authorization", "proxy-authenticate", + "te", "trailer", "upgrade", "host", "expect"}) +SITE_ROOT_HEADER = "AgentEnv-Site-Root" +FORWARDED = frozenset({"x-forwarded-host", "x-forwarded-proto", "x-forwarded-for", "x-forwarded-port", + "x-forwarded-prefix", "forwarded", SITE_ROOT_HEADER.lower()}) +RESPONSE_DROP = frozenset({"connection", "keep-alive", "proxy-connection", "alt-svc"}) +UPSTREAM_HEADERS_FILE = "upstream_headers.json" +HEAD_LIMIT = 1 << 20 +CHUNK = 1 << 16 +UPSTREAM_TIMEOUT_S = 30.0 +DNS_TIMEOUT_S = 4.0 +NAMESERVER_PORT = 53 +DATAGRAM_MAX = 65535 +LISTEN_TIMEOUT_S = 15.0 +CHECK_TIMEOUT_S = 15.0 +RESTART_DELAY_S = 1.0 +SERVE_USER = "nobody" +FAILED = (("X-Agentenv-Site-Proxy", "failed"),) + + +class BadRequest(Exception): + """A request the proxy can't read.""" + + +def is_local(name: str) -> bool: + """A ``.localhost`` name, served over HTTP: browsers treat it as secure without TLS.""" + return name == "localhost" or name.endswith(".localhost") + + +def matches(pattern: str, host: str) -> bool: + """Whether ``host`` is the name ``pattern`` names: itself, or for ``*.example.com`` one label more, as a + certificate's wildcard matches.""" + host = host.lower().rstrip(".") + if pattern.startswith("*."): + head, dot, rest = host.partition(".") + return bool(head) and bool(dot) and rest == pattern[2:] + return host == pattern + + +@dataclass +class Site: + """One site as the proxy serves it: its names, the route on the gateway it forwards to, and its certificate and key + files, for a site with a name served over HTTPS.""" + + name: str + upstream: str + domains: list[str] + cert: Optional[str] = None + key: Optional[str] = None + upstream_headers: dict[str, str] = field(default_factory=dict) + scheme: str = field(init=False) + host: str = field(init=False) + port: int = field(init=False) + prefix: str = field(init=False) + + def __post_init__(self) -> None: + parts = urlsplit(self.upstream) + if parts.scheme not in ("http", "https") or not parts.hostname: + raise ValueError(f"site {self.name}: its upstream {self.upstream!r} isn't an http(s) URL") + self.scheme, self.host = parts.scheme, parts.hostname + self.port = parts.port or (443 if parts.scheme == "https" else 80) + self.prefix = parts.path.rstrip("/") + + @property + def netloc(self) -> str: + default = 443 if self.scheme == "https" else 80 + return self.host if self.port == default else f"{self.host}:{self.port}" + + def tls(self, name: str) -> bool: + return not is_local(name) + + +@dataclass +class Config: + """What ``serve`` reads: the session user, the user it drops to (``serve_user``, else the session user), its + ports, the sites and, with a wildcard name, the nameservers it forwards every other question to.""" + + user: Optional[str] + sites: list[Site] + serve_user: Optional[str] = None + listen: str = LOOPBACK + https_port: int = 443 + http_port: int = 80 + dns_port: Optional[int] = None + nameservers: list[str] = field(default_factory=list) + + @classmethod + def load(cls, path: str) -> Config: + """The config at ``path``, with each site's upstream headers from the private file it names, which root alone + reads: ``serve`` reads it before it drops privileges, and ``check``, as the session user, goes without.""" + data = json.loads(Path(path).read_text()) + headers: dict = {} + if data.get("upstream_headers"): + try: + headers = json.loads(Path(data["upstream_headers"]).read_text()) + except PermissionError: + headers = {} + sites = [Site(name=s["name"], upstream=s["upstream"], domains=list(s["domains"]), cert=s.get("cert"), + key=s.get("key"), upstream_headers=dict(headers.get(s["name"]) or {})) for s in data["sites"]] + return cls(user=data.get("user"), sites=sites, serve_user=data.get("serve_user"), + listen=data.get("listen", LOOPBACK), + https_port=data.get("https_port", 443), http_port=data.get("http_port", 80), + dns_port=data.get("dns_port"), nameservers=list(data.get("nameservers") or [])) + + def site_for(self, host: str) -> Optional[Site]: + """The site ``host`` names: one naming it exactly before one whose wildcard covers it.""" + host = host.lower().rstrip(".") + for exact in (True, False): + for site in self.sites: + if any(matches(name, host) for name in site.domains if name.startswith("*.") != exact): + return site + return None + + +def header(headers: list[tuple[str, str]], name: str) -> Optional[str]: + return next((value for key, value in headers if key.lower() == name), None) + + +def tokens(headers: list[tuple[str, str]], name: str) -> set[str]: + return {token.strip().lower() for key, value in headers if key.lower() == name for token in value.split(",")} + + +def parse_head(data: bytes) -> tuple[str, list[tuple[str, str]]]: + """A message head's start line and its headers, in order.""" + lines = data.decode("latin-1").removesuffix("\r\n\r\n").split("\r\n") + headers = [] + for line in lines[1:]: + key, colon, value = line.partition(":") + if not colon or not key or key != key.strip(): + raise BadRequest(f"a malformed header line {line!r}") + headers.append((key, value.strip())) + return lines[0], headers + + +def split_host(host: str) -> str: + """The name of a ``Host`` value, without its port.""" + if host.startswith("["): + return host[1:].partition("]")[0] + return host.rpartition(":")[0] if host.count(":") == 1 else host + + +def origin(scheme: str, netloc: str) -> tuple[str, str, int]: + """A URL's scheme, name and port, its port the scheme's own when it names none.""" + parts = urlsplit(f"//{netloc}") + scheme = scheme.lower() + return scheme, (parts.hostname or "").lower(), parts.port or (443 if scheme == "https" else 80) + + +def public_location(location: str, site: Site, scheme: str, host: str) -> str: + """A ``Location`` the upstream sent, as the browser follows it: one on the site's route, under the gateway's address, + under the site's own name (as a site that ignores ``AgentEnv-Site-Root`` may send) or as a path, is rewritten onto + the site's name; any other is left alone. A scheme-relative one is read with the scheme of the page it was sent + to: the route's for the gateway's address, the browser's for the site's name.""" + parts = urlsplit(location) + if parts.netloc: + pages = ((site.scheme, site.netloc), (scheme, host)) + if not any(origin(parts.scheme or own, parts.netloc) == origin(own, netloc) for own, netloc in pages): + return location + elif parts.scheme or not location.startswith("/"): + return location + path = parts.path + if not (path == site.prefix or path.startswith(site.prefix + "/")) or not site.prefix: + return location + rest = path[len(site.prefix):] or "/" + origin_parts = (scheme, host) if parts.netloc else ("", "") + return urlunsplit((*origin_parts, rest, parts.query, parts.fragment)) + + +def public_cookie(value: str, site: Site) -> str: + """A ``Set-Cookie`` the upstream sent, with any ``Domain`` naming the gateway dropped and a ``Path`` under the route + rooted at ``/``, so the browser keeps it for the site's name.""" + first, *attributes = value.split(";") + kept = [first] + for attribute in attributes: + key, _, attr_value = attribute.strip().partition("=") + key, attr_value = key.strip(), attr_value.strip() + if key.lower() == "domain" and attr_value.strip().lstrip(".").lower() == site.host.lower(): + continue + if key.lower() == "path" and site.prefix and (attr_value == site.prefix + or attr_value.startswith(site.prefix + "/")): + attribute = f" Path={attr_value[len(site.prefix):] or '/'}" + kept.append(attribute) + return ";".join(kept) + + +def upstream_request(method: str, target: str, headers: list[tuple[str, str]], site: Site, *, scheme: str, + host: str, peer: str, upgrade: Optional[str]) -> bytes: + """The request head sent to the site's route: the path under its prefix, the gateway as ``Host``, the browser's + name and scheme in ``X-Forwarded-Host`` and ``X-Forwarded-Proto``, ``AgentEnv-Site-Root`` so the site is told it + is at ``/``, the headers the gateway's sandbox provider asks for (``Site.upstream_headers``) in place of any the + browser sent by those names, and a connection that closes after it, or upgrades when the browser asked to.""" + if not target.startswith("/"): + parts = urlsplit(target) + target = urlunsplit(("", "", parts.path or "/", parts.query, "")) + provided = {key.lower() for key in site.upstream_headers} + kept = [(key, value) for key, value in headers if key.lower() not in HOP_BY_HOP | FORWARDED | provided] + kept += [("Host", site.netloc), ("X-Forwarded-Host", host), ("X-Forwarded-Proto", scheme), + ("X-Forwarded-For", peer), (SITE_ROOT_HEADER, "1"), *site.upstream_headers.items()] + kept += [("Connection", "Upgrade"), ("Upgrade", upgrade)] if upgrade else [("Connection", "close")] + lines = [f"{method} {site.prefix}{target} HTTP/1.1", *(f"{key}: {value}" for key, value in kept)] + return ("\r\n".join(lines) + "\r\n\r\n").encode("latin-1") + + +def response_head(status_line: str, headers: list[tuple[str, str]], site: Site, *, scheme: str, host: str, + connection: str) -> bytes: + _, _, status = status_line.partition(" ") + kept = [] + for key, value in headers: + lower = key.lower() + if lower in RESPONSE_DROP or (lower == "upgrade" and connection != "Upgrade"): + continue + if lower == "location": + value = public_location(value, site, scheme, host) + elif lower == "set-cookie": + value = public_cookie(value, site) + kept.append((key, value)) + kept.append(("Connection", connection)) + lines = [f"HTTP/1.1 {status}", *(f"{key}: {value}" for key, value in kept)] + return ("\r\n".join(lines) + "\r\n\r\n").encode("latin-1") + + +def simple_response(status: str, body: str, extra: tuple[tuple[str, str], ...] = ()) -> bytes: + data = body.encode() + lines = [f"HTTP/1.1 {status}", "Content-Type: text/plain; charset=utf-8", f"Content-Length: {len(data)}", + *(f"{key}: {value}" for key, value in extra), "Connection: close"] + return ("\r\n".join(lines) + "\r\n\r\n").encode("latin-1") + data + + +async def read_head(reader: asyncio.StreamReader) -> Optional[bytes]: + """The next message head, or None when the connection closed before one began.""" + try: + return await reader.readuntil(b"\r\n\r\n") + except asyncio.IncompleteReadError as e: + if not e.partial.strip(): + return None + raise BadRequest("the connection closed mid-head") from None + except asyncio.LimitOverrunError: + raise BadRequest("a head over the limit") from None + + +async def relay_exact(reader: asyncio.StreamReader, writer: asyncio.StreamWriter, size: int) -> None: + while size > 0: + data = await reader.read(min(size, CHUNK)) + if not data: + raise BadRequest("the body ended early") + writer.write(data) + await writer.drain() + size -= len(data) + + +async def relay_chunked(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + """A chunked body relayed as framed, to its last chunk and trailers.""" + while True: + line = await reader.readuntil(b"\r\n") + writer.write(line) + try: + size = int(line.split(b";", 1)[0].strip(), 16) + except ValueError: + raise BadRequest(f"a malformed chunk size {line!r}") from None + if size == 0: + while (trailer := await reader.readuntil(b"\r\n")) != b"\r\n": + writer.write(trailer) + writer.write(trailer) + await writer.drain() + return + await relay_exact(reader, writer, size + 2) + + +async def relay_until_eof(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + while data := await reader.read(CHUNK): + writer.write(data) + await writer.drain() + + +async def relay_body(reader: asyncio.StreamReader, writer: asyncio.StreamWriter, headers: list[tuple[str, str]], + *, to_eof: bool) -> None: + if "chunked" in tokens(headers, "transfer-encoding"): + await relay_chunked(reader, writer) + elif (length := header(headers, "content-length")) is not None: + try: + await relay_exact(reader, writer, int(length)) + except ValueError: + raise BadRequest(f"a malformed Content-Length {length!r}") from None + elif to_eof: + await relay_until_eof(reader, writer) + + +async def splice(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + """One direction of an upgraded connection, closing the other side when it ends, which ends the other direction.""" + try: + await relay_until_eof(reader, writer) + except (ConnectionError, OSError, ssl.SSLError): + pass + finally: + writer.close() + + +class SiteProxy: + """The proxy's HTTP side: one connection's requests, each forwarded on a connection of its own to its site's + route.""" + + def __init__(self, config: Config): + self.config = config + self.upstream_tls = ssl.create_default_context() + + async def handle(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, scheme: str) -> None: + peer = (writer.get_extra_info("peername") or (LOOPBACK,))[0] + try: + while await self.exchange(reader, writer, scheme, peer): + pass + except BadRequest as e: + writer.write(simple_response("400 Bad Request", f"{e}\n")) + except (ConnectionError, asyncio.IncompleteReadError, ssl.SSLError, OSError): + pass + except Exception as e: + print(f"site proxy: a {scheme} connection failed: {type(e).__name__}: {e}", file=sys.stderr, flush=True) + finally: + try: + await writer.drain() + writer.close() + await writer.wait_closed() + except (ConnectionError, OSError, ssl.SSLError): + pass + + async def exchange(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, scheme: str, + peer: str) -> bool: + """One request and its answer; whether the connection takes another.""" + head = await read_head(reader) + if head is None: + return False + start, headers = parse_head(head) + try: + method, target, version = start.split(" ") + except ValueError: + raise BadRequest(f"a malformed request line {start!r}") from None + host = header(headers, "host") or (urlsplit(target).netloc if "://" in target else "") + site = self.config.site_for(split_host(host)) if host else None + if site is None: + writer.write(simple_response("421 Misdirected Request", f"{host or 'no host'} is no site of this " + "computer\n", FAILED)) + return False + if scheme == "http" and site.tls(split_host(host).lower()): + location = f"https://{split_host(host)}{target if target.startswith('/') else urlsplit(target).path}" + writer.write(simple_response("308 Permanent Redirect", f"{location}\n", (("Location", location),))) + return False + upgrade = header(headers, "upgrade") if "upgrade" in tokens(headers, "connection") else None + if "100-continue" in tokens(headers, "expect"): + writer.write(b"HTTP/1.1 100 Continue\r\n\r\n") + try: + up_reader, up_writer = await asyncio.wait_for(asyncio.open_connection( + site.host, site.port, ssl=self.upstream_tls if site.scheme == "https" else None, + server_hostname=site.host if site.scheme == "https" else None, limit=HEAD_LIMIT), UPSTREAM_TIMEOUT_S) + except (OSError, asyncio.TimeoutError, ssl.SSLError) as e: + writer.write(simple_response("502 Bad Gateway", f"the site {site.name} can't be reached at " + f"{site.upstream}: {type(e).__name__}: {e}\n", FAILED)) + return False + try: + return await self._forward(reader, writer, up_reader, up_writer, method, target, version, headers, site, + scheme=scheme, host=host, peer=peer, upgrade=upgrade) + finally: + up_writer.close() + + async def _forward(self, reader, writer, up_reader, up_writer, method, target, version, headers, site, *, + scheme, host, peer, upgrade) -> bool: + up_writer.write(upstream_request(method, target, headers, site, scheme=scheme, host=host, peer=peer, + upgrade=upgrade)) + await relay_body(reader, up_writer, headers, to_eof=False) + await up_writer.drain() + while True: + answer = await read_head(up_reader) + if answer is None: + writer.write(simple_response("502 Bad Gateway", f"the site {site.name} closed without answering\n", + FAILED)) + return False + status_line, answer_headers = parse_head(answer) + code = int(status_line.split(" ")[1]) + if code == 101 and upgrade: + writer.write(response_head(status_line, answer_headers, site, scheme=scheme, host=host, + connection="Upgrade")) + await writer.drain() + await asyncio.gather(splice(reader, up_writer), splice(up_reader, writer)) + return False + if 100 <= code < 200: + continue + break + bodiless = method == "HEAD" or code in (204, 304) + framed = bodiless or "chunked" in tokens(answer_headers, "transfer-encoding") \ + or header(answer_headers, "content-length") is not None + keep = framed and version == "HTTP/1.1" and "close" not in tokens(headers, "connection") + writer.write(response_head(status_line, answer_headers, site, scheme=scheme, host=host, + connection="keep-alive" if keep else "close")) + if not bodiless: + try: + await relay_body(up_reader, writer, answer_headers, to_eof=True) + except (BadRequest, asyncio.IncompleteReadError): + return False + await writer.drain() + return keep + + +class Resolver(asyncio.DatagramProtocol): + """DNS on loopback for a wildcard name, over UDP and over TCP, which a client retries a truncated answer on and + ``options use-vc`` asks for: each question for a name of a site is answered with loopback, and every other is passed + to the computer's own nameservers over the transport it came on, so the rest of the network is as it was; with none + (a computer whose resolv.conf named no other), every other question fails at once rather than going unanswered.""" + + def __init__(self, config: Config): + self.config = config + self.transport: Optional[asyncio.DatagramTransport] = None + + def connection_made(self, transport) -> None: + self.transport = transport + + def datagram_received(self, data: bytes, addr) -> None: + asyncio.ensure_future(self._answer(data, addr)) + + async def _answer(self, data: bytes, addr) -> None: + reply = await self.answer(data, stream=False) + if reply and self.transport is not None: + self.transport.sendto(reply, addr) + + async def handle_stream(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + """A TCP connection's queries, each framed by its two-byte length, answered in turn until it closes.""" + try: + while True: + try: + (size,) = struct.unpack("!H", await reader.readexactly(2)) + except asyncio.IncompleteReadError: + return + reply = await self.answer(await reader.readexactly(size), stream=True) + if not reply: + return + writer.write(struct.pack("!H", len(reply)) + reply) + await writer.drain() + except (ConnectionError, OSError, asyncio.IncompleteReadError): + pass + finally: + writer.close() + + async def answer(self, data: bytes, *, stream: bool) -> Optional[bytes]: + question = parse_question(data) + if question is not None and self.config.site_for(question[0]) is not None: + return local_answer(data, question) + if not self.config.nameservers: + return failed_answer(data, question) if question is not None else None + return await (self._passed_on_stream(data) if stream else self._passed_on(data)) + + async def _passed_on(self, data: bytes) -> Optional[bytes]: + loop = asyncio.get_running_loop() + for nameserver in self.config.nameservers: + family = socket.AF_INET6 if ":" in nameserver else socket.AF_INET + with socket.socket(family, socket.SOCK_DGRAM) as sock: + sock.setblocking(False) + try: + await loop.sock_connect(sock, (nameserver, NAMESERVER_PORT)) + await loop.sock_sendall(sock, data) + return await asyncio.wait_for(loop.sock_recv(sock, DATAGRAM_MAX), DNS_TIMEOUT_S) + except (OSError, asyncio.TimeoutError): + continue + return None + + async def _passed_on_stream(self, data: bytes) -> Optional[bytes]: + for nameserver in self.config.nameservers: + try: + return await asyncio.wait_for(self._ask_stream(nameserver, data), DNS_TIMEOUT_S) + except (OSError, asyncio.TimeoutError, asyncio.IncompleteReadError): + continue + return None + + @staticmethod + async def _ask_stream(nameserver: str, data: bytes) -> bytes: + reader, writer = await asyncio.open_connection(nameserver, NAMESERVER_PORT) + try: + writer.write(struct.pack("!H", len(data)) + data) + await writer.drain() + (size,) = struct.unpack("!H", await reader.readexactly(2)) + return await reader.readexactly(size) + finally: + writer.close() + + +def parse_question(data: bytes) -> Optional[tuple[str, int, int]]: + """A query's one question, as (name, type, where it ends), or None for anything else.""" + if len(data) < 12: + return None + _, flags, questions = struct.unpack("!HHH", data[:6]) + if flags & 0x8000 or questions != 1: + return None + labels, at = [], 12 + while at < len(data) and (length := data[at]): + if length & 0xC0: + return None + labels.append(data[at + 1:at + 1 + length].decode("ascii", "replace")) + at += 1 + length + if at + 5 > len(data): + return None + (qtype,) = struct.unpack("!H", data[at + 1:at + 3]) + return ".".join(labels).lower(), qtype, at + 5 + + +def local_answer(query: bytes, question: tuple[str, int, int]) -> bytes: + """The answer naming loopback for an A question about a site's name, and no records for any other type, so a + client asking for IPv6 falls back to it.""" + _, qtype, end = question + (ident, flags) = struct.unpack("!HH", query[:4]) + answers = 1 if qtype in (1, 255) else 0 + head = struct.pack("!HHHHHH", ident, 0x8480 | (flags & 0x0100), 1, answers, 0, 0) + record = struct.pack("!HHHIH", 0xC00C, 1, 1, 60, 4) + socket.inet_aton(LOOPBACK) if answers else b"" + return head + query[12:end] + record + + +def failed_answer(query: bytes, question: tuple[str, int, int]) -> bytes: + """The answer saying the server failed (SERVFAIL), for a question it has no nameserver to pass on to.""" + (ident, flags) = struct.unpack("!HH", query[:4]) + return struct.pack("!HHHHHH", ident, 0x8082 | (flags & 0x0100), 1, 0, 0, 0) + query[12:question[2]] + + +def tls_context(config: Config) -> Optional[ssl.SSLContext]: + """The HTTPS side's context: each site's certificate, picked by the name the browser asks for.""" + contexts = {} + for site in config.sites: + if site.cert and site.key: + context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + context.minimum_version = ssl.TLSVersion.TLSv1_2 + context.set_alpn_protocols(["http/1.1"]) + context.load_cert_chain(site.cert, site.key) + contexts[site.name] = context + if not contexts: + return None + default = next(iter(contexts.values())) + + def pick(connection: ssl.SSLObject, name: Optional[str], _: ssl.SSLContext) -> None: + site = config.site_for(name) if name else None + if site is not None and site.name in contexts: + connection.context = contexts[site.name] + + default.sni_callback = pick + return default + + +async def serve(config: Config) -> None: + """Listen, drop to the user it serves as, and serve until cancelled.""" + proxy = SiteProxy(config) + servers = [] + if (context := tls_context(config)) is not None: + servers.append(await asyncio.start_server(lambda r, w: proxy.handle(r, w, "https"), config.listen, + config.https_port, ssl=context, limit=HEAD_LIMIT)) + servers.append(await asyncio.start_server(lambda r, w: proxy.handle(r, w, "http"), config.listen, + config.http_port, limit=HEAD_LIMIT)) + transport = None + if config.dns_port: + resolver = Resolver(config) + transport, _ = await asyncio.get_running_loop().create_datagram_endpoint( + lambda: resolver, local_addr=(config.listen, config.dns_port)) + servers.append(await asyncio.start_server(resolver.handle_stream, config.listen, config.dns_port)) + drop_privileges(config.serve_user or config.user) + try: + await asyncio.gather(*(server.serve_forever() for server in servers)) + finally: + if transport is not None: + transport.close() + + +def serve_user(session_user: Optional[str]) -> Optional[str]: + """The user ``serve`` drops to: ``SERVE_USER`` where the base has it, else the session user.""" + try: + pwd.getpwnam(SERVE_USER) + except KeyError: + return session_user + return SERVE_USER + + +def drop_privileges(user: Optional[str]) -> None: + if user is None or os.geteuid() != 0: + return + entry = pwd.getpwnam(user) + os.setgroups([]) + os.setgid(entry.pw_gid) + os.setuid(entry.pw_uid) + + +@dataclass(frozen=True) +class Paths: + """Where ``install`` writes, under ``root``, by the conventions of Debian-family and Red Hat-family Linux.""" + + root: Path = Path("/") + + @property + def hosts(self) -> Path: + return self.root / "etc/hosts" + + @property + def resolv(self) -> Path: + return self.root / "etc/resolv.conf" + + @property + def anchors(self) -> tuple[tuple[Path, list[str]], ...]: + """Each system store's anchor file, with the tool that rebuilds the store from its anchors.""" + return ((self.root / f"usr/local/share/ca-certificates/{CA_NAME}.crt", ["update-ca-certificates"]), + (self.root / f"etc/pki/ca-trust/source/anchors/{CA_NAME}.pem", ["update-ca-trust", "extract"])) + + @property + def firefox_policies(self) -> Path: + return self.root / "etc/firefox/policies/policies.json" + + @property + def chromium_policies(self) -> tuple[Path, ...]: + """Where each Chromium build reads managed policies: Chrome for Testing, Chromium, a snap's Chromium and + Chrome.""" + return tuple(self.root / folder / f"policies/managed/{CA_NAME}.json" + for folder in ("etc/opt/chrome_for_testing", "etc/chromium", "etc/chromium-browser", + "etc/opt/chrome")) + + @property + def profile(self) -> Path: + return self.root / f"etc/profile.d/{CA_NAME}.sh" + + @property + def system_bundle(self) -> Path: + return self.root / "etc/ssl/certs/ca-certificates.crt" + + +def hosts_text(current: str, names: list[str]) -> str: + """``current``, a hosts file, with its block of the sites' names replaced by one mapping ``names`` to loopback.""" + lines, inside = [], False + for line in current.splitlines(): + if line.strip() == HOSTS_BEGIN: + inside = True + elif line.strip() == HOSTS_END: + inside = False + elif not inside: + lines.append(line) + if names: + lines += [HOSTS_BEGIN, *(f"{LOOPBACK} {name}" for name in names), HOSTS_END] + return "\n".join(lines) + "\n" + + +def resolv_text(current: str) -> tuple[str, list[str]]: + """``current``, a resolv.conf, asking loopback first, and the nameservers it named, which loopback passes the rest + to.""" + nameservers = [line.split()[1] for line in current.splitlines() + if line.split()[:1] == ["nameserver"] and len(line.split()) > 1] + passed = [ns for ns in nameservers if ns != LOOPBACK] + kept = [line for line in current.splitlines() if line.split()[:1] != ["nameserver"]] + return "\n".join([f"nameserver {LOOPBACK}", *kept]) + "\n", passed + + +def firefox_policies(current: Optional[str], ca: str) -> str: + """Firefox's policies file with ``ca`` among the certificates it installs, every other policy kept.""" + data = json.loads(current) if current and current.strip() else {} + installs = data.setdefault("policies", {}).setdefault("Certificates", {}).setdefault("Install", []) + if ca not in installs: + installs.append(ca) + return json.dumps(data, indent=2) + "\n" + + +def chromium_policies(ca_pem: str) -> str: + """A Chromium managed policy trusting ``ca_pem`` (``CACertificates``, base64 DER, which is a PEM's body), which + Chromium reads without ``certutil`` or an NSS database.""" + body = "".join(line.strip() for line in ca_pem.splitlines() if line.strip() and not line.startswith("-----")) + return json.dumps({"CACertificates": [body]}, indent=1) + "\n" + + +def write_private(path: Path, text: str) -> None: + """Write ``path`` readable by its owner alone, from its first byte.""" + with os.fdopen(os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), "w") as f: + f.write(text) + path.chmod(0o600) + + +def write_in_place(path: Path, text: str) -> None: + """Rewrite ``path`` in place, which a file a container's runtime mounts (its hosts and resolv.conf) needs.""" + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "w") as f: + f.write(text) + + +def trust_system(paths: Paths, ca_pem: str) -> Optional[str]: + """Add the CA to the system store, by the tool this Linux has; which store, or None when it has neither.""" + for anchor, tool in paths.anchors: + if shutil.which(tool[0]): + write_in_place(anchor, ca_pem) + subprocess.run(tool, check=True, capture_output=True) + return str(anchor) + return None + + +def trust_nss(user: Optional[str], ca: str) -> list[str]: + """Add the CA to the session user's NSS databases, Chromium's (and a snap Chromium's when there is one), as that + user, for a Chromium too old to read its ``CACertificates`` policy; none without ``certutil``.""" + if user is None or not shutil.which("certutil"): + return [] + home = Path(pwd.getpwnam(user).pw_dir) + folders = [home / ".pki" / "nssdb"] + if (home / "snap" / "chromium").is_dir(): + folders.append(home / "snap" / "chromium" / "current" / ".pki" / "nssdb") + trusted = [] + for folder in folders: + db = f"sql:{folder}" + steps = [["mkdir", "-p", str(folder)]] + if not (folder / "cert9.db").exists(): + steps.append(["certutil", "-N", "-d", db, "--empty-password"]) + steps.append(["certutil", "-A", "-d", db, "-n", NSS_NICKNAME, "-t", "C,,", "-i", ca]) + for step in steps: + subprocess.run(as_user(user, step), check=True, capture_output=True) + trusted.append(str(folder)) + return trusted + + +def as_user(user: Optional[str], argv: list[str]) -> list[str]: + if user is None or os.geteuid() != 0: + return argv + return ["runuser", "-u", user, "--", *argv] + + +def install(config_path: str, paths: Paths = Paths()) -> dict: + """As root: write the sites' certificates, names and trust, start ``serve`` on what's written, and check each name + loads as the session user; what was trusted and each check's result. The config's folder, which arrives root's + alone, is opened to the session user (for the CA) only once its keys and the config are private.""" + if os.geteuid() != 0 and paths.root == Path("/"): + raise SystemExit(f"the site proxy installs as root, not uid {os.geteuid()}") + given = json.loads(Path(config_path).read_text()) + folder = Path(config_path).parent + user = given.get("user") + ca = str(folder / "ca.pem") + if given.get("ca_pem"): + Path(ca).write_text(given["ca_pem"]) + sites = [] + for site in given["sites"]: + served = {"name": site["name"], "upstream": site["upstream"], "domains": site["domains"]} + if site.get("cert_pem"): + cert, key = folder / f"{site['name']}.crt", folder / f"{site['name']}.key" + cert.write_text(site["cert_pem"]) + write_private(key, site["key_pem"]) + served.update(cert=str(cert), key=str(key)) + sites.append(served) + upstream_headers = {site["name"]: site["upstream_headers"] for site in given["sites"] if site.get("upstream_headers")} + if upstream_headers: + write_private(folder / UPSTREAM_HEADERS_FILE, json.dumps(upstream_headers)) + os.chmod(config_path, 0o600) + folder.chmod(0o755) + names = [name for site in given["sites"] for name in site["domains"] if not name.startswith("*.")] + write_in_place(paths.hosts, hosts_text(paths.hosts.read_text() if paths.hosts.exists() else "", names)) + trust: dict = {} + nameservers: list[str] = [] + wildcard = any(name.startswith("*.") for site in given["sites"] for name in site["domains"]) + if wildcard: + text, nameservers = resolv_text(paths.resolv.read_text() if paths.resolv.exists() else "") + write_in_place(paths.resolv, text) + if given.get("ca_pem"): + trust["system"] = trust_system(paths, given["ca_pem"]) + write_in_place(paths.firefox_policies, firefox_policies( + paths.firefox_policies.read_text() if paths.firefox_policies.exists() else None, ca)) + trust["firefox"] = str(paths.firefox_policies) + for policy in paths.chromium_policies: + write_in_place(policy, chromium_policies(given["ca_pem"])) + trust["chromium"] = [str(policy) for policy in paths.chromium_policies] + trust["nss"] = trust_nss(user, ca) + bundle = paths.system_bundle if paths.system_bundle.exists() else None + write_in_place(paths.profile, f"export NODE_EXTRA_CA_CERTS={ca}\n" + + (f"export REQUESTS_CA_BUNDLE={bundle}\n" if bundle else "")) + trust["environment"] = str(paths.profile) + serving = {"user": user, "serve_user": serve_user(user), "sites": sites, **{key: given[key] for key in ("listen", "https_port", "http_port") + if key in given}} + if wildcard: + serving.update(dns_port=given.get("dns_port", 53), nameservers=nameservers) + if upstream_headers: + serving["upstream_headers"] = str(folder / UPSTREAM_HEADERS_FILE) + serve_path = folder / "serve.json" + serve_path.write_text(json.dumps(serving)) + start(str(serve_path), given.get("log")) + return {"trust": trust, "names": names, "checks": checks_as(user, str(serve_path))} + + +def start(serve_path: str, log: Optional[str]) -> None: + """Start ``keep``, which runs ``serve``, apart from this process, and wait until it listens. The log is opened + without following a link, so a link planted at its path can't have root write elsewhere.""" + config = Config.load(serve_path) + output = open_log(log) if log else subprocess.DEVNULL + subprocess.Popen([sys.executable, os.path.abspath(__file__), "keep", serve_path], stdin=subprocess.DEVNULL, + stdout=output, stderr=subprocess.STDOUT, start_new_session=True, close_fds=True) + port = config.https_port if any(site.cert for site in config.sites) else config.http_port + deadline = time.monotonic() + LISTEN_TIMEOUT_S + while True: + try: + socket.create_connection((config.listen, port), timeout=1).close() + return + except OSError: + if time.monotonic() > deadline: + tail = Path(log).read_text()[-2000:] if log and Path(log).exists() else "" + raise SystemExit(f"the site proxy didn't listen on {config.listen}:{port} within " + f"{LISTEN_TIMEOUT_S:.0f} s: {tail}") + time.sleep(0.2) + + +def open_log(log: str) -> int: + Path(log).parent.mkdir(parents=True, exist_ok=True) + return os.open(log, os.O_WRONLY | os.O_APPEND | os.O_CREAT | os.O_NOFOLLOW, 0o600) + + +def keep(serve_path: str) -> None: + """As root: run ``serve`` on ``serve_path``, and start it again ``RESTART_DELAY_S`` after it ends, for as long as + this runs.""" + while True: + ended = subprocess.run([sys.executable, os.path.abspath(__file__), "serve", serve_path], + stdin=subprocess.DEVNULL).returncode + print(f"the site proxy ended ({ended}); starting it again", flush=True) + time.sleep(RESTART_DELAY_S) + + +def checks_as(user: Optional[str], serve_path: str) -> dict: + """``check`` run as ``user``, given each site's timeout one after another and a margin for its start.""" + sites = len(Config.load(serve_path).sites) + done = subprocess.run(as_user(user, [sys.executable, os.path.abspath(__file__), "check", serve_path]), + capture_output=True, text=True, timeout=CHECK_TIMEOUT_S * (sites + 2)) + if done.returncode != 0: + return {"*": {"error": (done.stderr.strip().splitlines() or ["the check failed"])[-1]}} + return json.loads(done.stdout) + + +def check(config: Config) -> dict: + """Each site's first name loaded, over HTTPS but for a ``.localhost`` name, through the system's resolver and + trust store: its status, or why it didn't load, which an answer the proxy made itself (it marks them ``FAILED``), + such as its 502 for a gateway it couldn't reach, is too.""" + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + results = {} + for site in config.sites: + name = next((n for n in site.domains if not n.startswith("*.")), None) \ + or "agentenv-check." + site.domains[0].removeprefix("*.") + url = f"{'https' if site.tls(name) else 'http'}://{name}/" + try: + with opener.open(url, timeout=CHECK_TIMEOUT_S) as response: + results[name] = {"url": url, "status": response.status} + except urllib.error.HTTPError as e: + if e.headers is not None and e.headers.get(FAILED[0][0]) == FAILED[0][1]: + body = e.read().decode(errors="replace").strip() + results[name] = {"url": url, "error": f"{e.code} {e.reason}: {body}"} + else: + results[name] = {"url": url, "status": e.code} + except (urllib.error.URLError, OSError, ssl.SSLError) as e: + results[name] = {"url": url, "error": str(getattr(e, "reason", e))} + return results + + +def main(argv: list[str]) -> None: + command, path = argv[0], argv[1] + if command == "install": + print(json.dumps(install(path))) + elif command == "keep": + keep(path) + elif command == "serve": + asyncio.run(serve(Config.load(path))) + elif command == "check": + print(json.dumps(check(Config.load(path)))) + else: + raise SystemExit(f"usage: site_proxy.py install|keep|serve|check , not {command!r}") + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/src/agent_env/env/envs/computer/utils/sites.py b/src/agent_env/env/envs/computer/utils/sites.py new file mode 100644 index 000000000..a07499913 --- /dev/null +++ b/src/agent_env/env/envs/computer/utils/sites.py @@ -0,0 +1,504 @@ +"""Websites in the computer: the browser in a computer opens a site by the names it declares (``https://notes.test/``) +and reaches it on the env gateway, at the site's route (``/website/notes/``), with TLS it trusts, while every +other host follows the computer's network. + +A deploy with ``env_params.site_routing`` is handed the run's deployed websites (``deployed_sites``), as ``sites``. Each +name is claimed once (``site_table``): a synthetic site's names are routed, a live site's and the network's allowlist +are left to the network, and a synthetic name wins over a live one, which wins over the allowlist; two sites of one +kind claiming a name are refused. Once the computer is booted, a CA made for the deploy signs a certificate for each +site's names (``issue``), and the site proxy (``site_proxy.py``) is installed through the sandbox's root channel and +started (``route``), which a reset does again on its fresh computer. In the computer they live in +``/etc/agentenv/sites``, readable by root alone but the CA, ``ca.pem``, at a fixed path for an app with a certificate +list of its own: the folder is root's alone (0700) before the config and its keys are written into it, until the +install has made the keys and the config private; the CA and certificates are valid for the deploy's ttl and a day more, from a few minutes back for a +clock that's behind.""" + +from __future__ import annotations + +import asyncio +import ipaddress +import json +import logging +import re +import shlex +from dataclasses import dataclass, field +from datetime import UTC, datetime, timedelta +from pathlib import Path, PurePosixPath +from typing import TYPE_CHECKING, Iterable, Optional +from urllib.parse import urlsplit + +import httpx +from cryptography import x509 +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.asymmetric import ec +from cryptography.x509.oid import ExtendedKeyUsageOID, NameOID + +from agent_env.env.envs.computer.utils.site_proxy import is_local, matches +from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy, VmSandbox + +if TYPE_CHECKING: + from agent_env.env.env import DeployedEnv + from agent_env.env.envs.computer.utils.params import Site + from agent_env.providers.sandbox_providers.sandbox import Sandbox + +FOLDER = PurePosixPath("/etc/agentenv/sites") +PROXY = FOLDER / "site_proxy.py" +CONFIG = FOLDER / "config.json" +CA_PATH = FOLDER / "ca.pem" +LOG = FOLDER / "site-proxy.log" +SYNTHETIC, LIVE, ALLOWLIST = "synthetic", "live", "allowlist" +RANKS = (SYNTHETIC, LIVE, ALLOWLIST) +VALID_MARGIN = timedelta(days=1) +CLOCK_SKEW = timedelta(minutes=5) +CARD_ATTEMPTS = 3 +CARD_RETRY_S = 1.0 +_LABEL = re.compile(r"^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$") +_PROXY_SOURCE = Path(__file__).with_name("site_proxy.py") +logger = logging.getLogger(__name__) + + +@dataclass(frozen=True) +class Claim: + """A name and who claims it: a site, or the network's allowlist, at its rank.""" + + name: str + rank: str + owner: str + + +@dataclass(frozen=True) +class Route: + """A synthetic site as the computer serves it: its names, and its route on the gateway as the computer reaches + it.""" + + site: str + url: str + domains: tuple[str, ...] + + @property + def tls_names(self) -> tuple[str, ...]: + return tuple(name for name in self.domains if not is_local(name)) + + +@dataclass +class SiteTable: + """Every name of the deployment: the routes, the names left to the network, and each claim a stronger one + shadows.""" + + routes: list[Route] = field(default_factory=list) + unrouted: dict[str, str] = field(default_factory=dict) + shadowed: list[tuple[Claim, Claim]] = field(default_factory=list) + + def to_dict(self) -> dict: + return {"routes": {route.site: list(route.domains) for route in self.routes}, "unrouted": self.unrouted, + "shadowed": [{"name": weak.name, "rank": weak.rank, "owner": weak.owner, "by": strong.owner} + for weak, strong in self.shadowed]} + + +class SiteConflict(ValueError): + """Two sites of one kind claim a name.""" + + +def name_problem(name: str) -> Optional[str]: + """Why ``name`` can't be a site's name, or None: a DNS name of letters, digits and hyphens, a wildcard only as its + whole first label, and no address.""" + if name != name.lower() or name.endswith("."): + return f"{name!r} must be lowercase, with no trailing dot" + try: + ipaddress.ip_address(name) + return f"{name!r} is an address, not a name" + except ValueError: + pass + labels = name.split(".") + if labels[0] == "*": + labels = labels[1:] + if len(labels) < 2: + return f"{name!r} is a wildcard over a single label" + if len(name) > 253 or not labels or not all(_LABEL.match(label) for label in labels): + return f"{name!r} isn't a DNS name (letters, digits and hyphens, a '*' only as its whole first label)" + return None + + +def overlap(a: str, b: str) -> bool: + """Whether a host could be named by both ``a`` and ``b``, a wildcard covering one label.""" + if a == b: + return True + for wild, other in ((a, b), (b, a)): + if wild.startswith("*.") and not other.startswith("*.") and other.partition(".")[2] == wild[2:]: + return True + return False + + +def covers(strong: str, weak: str) -> bool: + """Whether every host ``weak`` names is one ``strong`` names.""" + return strong == weak or (strong.startswith("*.") and not weak.startswith("*.") + and weak.partition(".")[2] == strong[2:]) + + +def fallback_name(site: str) -> str: + """The name a synthetic site declaring none is reached by: its name on ``.localhost``, lowercase, each ``_`` a + ``-``, and no ``-`` at either end of the label (``My_Site`` is ``my-site.localhost``). Two sites it gives one name + are refused as any two claiming it.""" + return f"{site.lower().replace('_', '-')[:63].strip('-')}.localhost" + + +def claims(sites: Iterable[Site], allow_hosts: Iterable[str] = ()) -> list[Claim]: + """Each name claimed: a synthetic site's, its own or ``fallback_name`` when it declares none; a live site's and the + hosts it calls; and the network's allowlist.""" + found = [] + for site in sites: + if site.live: + found += [Claim(name, LIVE, site.name) for name in dict.fromkeys([*site.domains, *site.allow_hosts])] + else: + found += [Claim(name, SYNTHETIC, site.name) + for name in dict.fromkeys(site.domains or [fallback_name(site.name)])] + return found + [Claim(host.lower(), ALLOWLIST, "network") for host in dict.fromkeys(allow_hosts)] + + +def site_table(sites: list[Site], allow_hosts: Iterable[str] = ()) -> SiteTable: + """The deployment's names: a synthetic site's routed, the rest left to the network but those a stronger claim + covers, a stronger claim shadowing each weaker one that overlaps it. Raises SiteConflict naming every name two sites + of one kind claim, and ValueError for a name that can't be one or a site named twice.""" + every = claims(sites, allow_hosts) + names = [site.name for site in sites] + invalid = [f"{claim.owner}: {problem}" for claim in every if claim.rank != ALLOWLIST + and (problem := name_problem(claim.name))] + invalid += [f"two sites are named {name!r}" for name in dict.fromkeys(names) if names.count(name) > 1] + invalid += [f"{name!r} isn't a site's name (letters, digits, '-' and '_')" for name in names + if not re.fullmatch(r"[A-Za-z0-9_-]+", name)] + if invalid: + raise ValueError("; ".join(invalid)) + conflicts, kept, table = [], [], SiteTable() + for rank in RANKS: + for claim in (c for c in every if c.rank == rank): + same = [k for k in kept if k.rank == rank and k.owner != claim.owner and overlap(k.name, claim.name)] + conflicts += [f"{claim.name!r} ({claim.owner}) overlaps {k.name!r} ({k.owner})" for k in same] + stronger = [k for k in kept if k.rank != rank and overlap(k.name, claim.name)] + if stronger := sorted(stronger, key=lambda k: not covers(k.name, claim.name))[:1]: + table.shadowed.append((claim, stronger[0])) + kept.append(claim) + if conflicts: + raise SiteConflict("names claimed twice: " + "; ".join(conflicts)) + by_site = {site.name: site for site in sites} + for owner in dict.fromkeys(c.owner for c in kept if c.rank == SYNTHETIC): + table.routes.append(Route(owner, by_site[owner].url, + tuple(c.name for c in kept if c.owner == owner and c.rank == SYNTHETIC))) + covered = {weak for weak, strong in table.shadowed if covers(strong.name, weak.name)} + table.unrouted = {c.name: c.rank for c in kept if c.rank != SYNTHETIC and c not in covered} + return table + + +def table_problems(sites: list[Site], allow_hosts: Iterable[str] = ()) -> list[str]: + try: + site_table(sites, allow_hosts) + except ValueError as e: + return [f"env_params.sites: {e}"] + return [] + + +@dataclass(frozen=True) +class Issued: + """The deploy's CA certificate, and each site's certificate and key, all PEM; no CA when no name is served over + HTTPS. The CA's key signs them and is gone: nothing more can be issued under it.""" + + ca_pem: Optional[str] + leaves: dict[str, tuple[str, str]] + + +def issue(routes: list[Route], *, valid_for: timedelta, label: str, now: Optional[datetime] = None) -> Issued: + """A CA name-constrained to the routes' names, so it vouches for nothing else however widely it's trusted, and a + certificate signed by it for each route's names served over HTTPS, all valid ``valid_for`` and a margin more.""" + tls = [route for route in routes if route.tls_names] + if not tls: + return Issued(None, {}) + now = now or datetime.now(UTC) + start, end = now - CLOCK_SKEW, now + valid_for + VALID_MARGIN + ca_key = ec.generate_private_key(ec.SECP256R1()) + subject = x509.Name([x509.NameAttribute(NameOID.ORGANIZATION_NAME, "agentenv"), + x509.NameAttribute(NameOID.COMMON_NAME, f"agentenv sites {label}"[:64])]) + permitted = list(dict.fromkeys(name.removeprefix("*.") for route in tls for name in route.tls_names)) + ca = ( + x509.CertificateBuilder() + .subject_name(subject).issuer_name(subject) + .public_key(ca_key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(start).not_valid_after(end) + .add_extension(x509.BasicConstraints(ca=True, path_length=0), critical=True) + .add_extension(_key_usage(key_cert_sign=True, crl_sign=True), critical=True) + .add_extension(x509.NameConstraints(permitted_subtrees=[x509.DNSName(n) for n in permitted], + excluded_subtrees=None), critical=True) + .add_extension(x509.SubjectKeyIdentifier.from_public_key(ca_key.public_key()), critical=False) + .sign(ca_key, hashes.SHA256()) + ) + leaves = {} + for route in tls: + key = ec.generate_private_key(ec.SECP256R1()) + cert = ( + x509.CertificateBuilder() + .subject_name(x509.Name([])) + .issuer_name(subject) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(start).not_valid_after(end) + .add_extension(x509.SubjectAlternativeName([x509.DNSName(n) for n in route.tls_names]), critical=True) + .add_extension(x509.BasicConstraints(ca=False, path_length=None), critical=True) + .add_extension(_key_usage(digital_signature=True), critical=True) + .add_extension(x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH]), critical=False) + .add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False) + .add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(ca_key.public_key()), critical=False) + .sign(ca_key, hashes.SHA256()) + ) + leaves[route.site] = (cert.public_bytes(serialization.Encoding.PEM).decode(), key.private_bytes( + serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption()).decode()) + return Issued(ca.public_bytes(serialization.Encoding.PEM).decode(), leaves) + + +def _key_usage(**granted: bool) -> x509.KeyUsage: + usages = ("digital_signature", "content_commitment", "key_encipherment", "data_encipherment", "key_agreement", + "key_cert_sign", "crl_sign", "encipher_only", "decipher_only") + return x509.KeyUsage(**{usage: granted.get(usage, False) for usage in usages}) + + +def proxy_config(table: SiteTable, issued: Issued, *, user: Optional[str], + upstream_headers: Optional[dict[str, dict[str, str]]] = None) -> dict: + """What ``site_proxy.py install`` reads: the session user, the CA, and each route with its certificate and key, and + the headers its gateway's sandbox provider asks requests to it to carry (``upstream_headers``, by site), which the + proxy sends upstream alone.""" + sites = [] + for route in table.routes: + site = {"name": route.site, "upstream": route.url, "domains": list(route.domains)} + if route.site in issued.leaves: + site["cert_pem"], site["key_pem"] = issued.leaves[route.site] + if (upstream_headers or {}).get(route.site): + site["upstream_headers"] = dict(upstream_headers[route.site]) + sites.append(site) + return {"user": user, "ca_pem": issued.ca_pem, "sites": sites, "log": str(LOG)} + + +def reachable_problem(policy: Optional[NetworkPolicy], routes: list[Route]) -> Optional[str]: + """Why ``policy``, the computer's network as its sandbox applies it, keeps it from a route's gateway, or None: a + gateway is reachable by its name in the allowed hosts, exactly or under a ``*.domain`` entry, or by an address in an + allowed range.""" + if policy is None or not policy.restricts_egress: + return None + hosts = sorted({urlsplit(route.url).hostname or "" for route in routes}) + blocked = [host for host in hosts if not _allows(policy, host)] + if blocked: + allowed = [*policy.allow_hosts, *policy.allow_cidrs] + return (f"the computer's network allows only {', '.join(allowed) or 'nothing'}, so it can't reach " + f"the gateway at {', '.join(blocked)} its sites are on") + return None + + +def gateway_problem(routes: list[Route]) -> Optional[str]: + """Why the computer couldn't reach a route's gateway once its sites' names are routed, or None: a gateway named by a + route's name, exactly or under its wildcard, would resolve to the proxy itself.""" + hosts = {urlsplit(route.url).hostname or "" for route in routes} + hidden = sorted({f"{host} (named by {route.site}'s {name})" for host in hosts for route in routes + for name in route.domains if matches(name, host)}) + if hidden: + return f"the gateway at {', '.join(hidden)} would resolve to the computer's own site proxy" + return None + + +def _allows(policy: NetworkPolicy, host: str) -> bool: + host = host.lower().rstrip(".") + if any(host == allowed or (allowed.startswith("*.") and host.endswith(allowed[1:])) + for allowed in (entry.lower().rstrip(".") for entry in policy.allow_hosts)): + return True + try: + address = ipaddress.ip_address(host) + except ValueError: + return False + for entry in (*policy.allow_cidrs, *policy.allow_hosts): + try: + if address in ipaddress.ip_network(entry, strict=False): + return True + except ValueError: + continue + return False + + +class RootChannel: + """Root inside a booted computer, through its sandbox: on a VM sandbox ``docker exec -u 0`` into its container, + elsewhere the sandbox's own exec, which must run as root, as a container sandbox running the image as its own user + (``USER user``) doesn't: that's refused before anything is written. The driver's computer-exec/v1 runs only as the + session user.""" + + def __init__(self, sandbox: Sandbox): + self.sandbox = sandbox + self.checked = False + + async def private_folder(self, path: PurePosixPath) -> None: + """Make ``path`` a folder root alone can enter, an existing one too, so what's written into it next is never + the session user's to read, whatever mode a file lands with.""" + await self.run(["mkdir", "-p", str(path)]) + await self.run(["chmod", "0700", str(path)]) + + async def write(self, path: PurePosixPath, text: str) -> None: + if isinstance(self.sandbox, VmSandbox): + await self.sandbox.write_file_from_text(text, str(path)) + return + await self.run(["mkdir", "-p", str(path.parent)]) + await self.sandbox.write_file_from_text(text, str(path)) + + async def write_secret(self, path: PurePosixPath, text: str) -> None: + """Write ``text``, which holds keys or credentials, to ``path``. On a VM it's staged on the host in a folder + its owner alone can open (``secret_dir``), written over the exec's stdin (``write_secret_file``), so it's + never on a command line, and moved into the container, its staged copy removed. Other sandboxes write files + through their provider's API, as ``write`` does.""" + if not isinstance(self.sandbox, VmSandbox): + await self.write(path, text) + return + await self.run(["mkdir", "-p", str(path.parent)]) + async with self.sandbox.secret_dir() as secrets: + staged = f"{secrets}/{path.name}" + await self.sandbox.write_secret_file(text.encode(), staged) + await self.sandbox.docker_cp(staged, f"{self.sandbox.container_name}:{path}", remove_source=True) + + async def run(self, argv: list[str]) -> str: + if isinstance(self.sandbox, VmSandbox): + return await self.sandbox.exec_script(f"docker exec -u 0 {shlex.quote(self.sandbox.container_name)} " + f"{shlex.join(argv)}") + if not self.checked: + uid = (await self._exec(["id", "-u"])).strip() + if uid != "0": + raise RuntimeError(f"site routing installs as root, and the {getattr(self.sandbox, 'type', 'given')} " + f"sandbox runs the computer's commands as uid {uid}; deploy it on a VM sandbox") + self.checked = True + return await self._exec(argv) + + async def _exec(self, argv: list[str]) -> str: + code, stdout, stderr = await self.sandbox.exec_with_output(*argv) + if code != 0: + raise RuntimeError(f"{shlex.join(argv)} failed (exit {code}):\nstdout: {stdout[-1500:]}\n" + f"stderr: {stderr[-1500:]}") + return stdout + + +async def route(sandbox: Sandbox, sites: list[Site], *, user: Optional[str], network: Optional[dict], + valid_for: timedelta, label: str) -> dict: + """Have the computer on ``sandbox`` reach each of ``sites`` by its names: the table made, the CA and certificates + issued, the proxy installed and started as root, and each name checked as ``user``; what the deploy record keeps of + it. Raises naming a name that didn't load.""" + requested = NetworkPolicy.from_dict(network) if network else None + table = site_table(sites, requested.allow_hosts if requested else ()) + table.routes = [Route(r.site, sandbox.url_from_sandbox(r.url), r.domains) for r in table.routes] + applied = getattr(sandbox, "network_policy", None) or requested + if (problem := reachable_problem(applied, table.routes) or gateway_problem(table.routes)) is not None: + raise RuntimeError(f"site routing: {problem}") + if not table.routes: + return {"names": table.to_dict()} + from agent_env.providers.sandbox_providers.sandbox_provider import sandbox_request_headers_for_url + + issued = issue(table.routes, valid_for=valid_for, label=label) + upstream_headers = {r.site: sandbox_request_headers_for_url(r.url) for r in table.routes} + channel = RootChannel(sandbox) + await channel.private_folder(FOLDER) + await channel.write(PROXY, _PROXY_SOURCE.read_text()) + await channel.write_secret(CONFIG, json.dumps(proxy_config(table, issued, user=user, + upstream_headers=upstream_headers))) + await channel.run(["chmod", "0600", str(CONFIG)]) + output = await channel.run(["python3", str(PROXY), "install", str(CONFIG)]) + report = json.loads(output.strip().splitlines()[-1]) + failed = {name: result for name, result in report["checks"].items() if "error" in result} + if failed: + raise RuntimeError("site routing: " + "; ".join(f"{result.get('url', name)} didn't load in the computer: " + f"{result['error']}" for name, result in failed.items())) + return {"names": table.to_dict(), "ca": issued.ca_pem, "ca_path": str(CA_PATH), "trust": report["trust"], + "checks": report["checks"]} + + +async def deployed_sites(deployed_envs: list[DeployedEnv], only: Optional[list[str]] = None) -> list[dict]: + """The websites a run has deployed, as ``env_params.sites`` names them: each ``website/v1`` child env of a deployed + env's stored card, with the route its composer serves it to a browser at (its params' ``browse_url``, joined onto + the env's address), and the names and liveness its params declare. A record whose card names no ``browse_url``, as + one a gateway made before it did, takes the route from its ``website_frontend_urls``. A site those name that the + stored card doesn't hold, every site of a record without a card or one whose card the gateway couldn't read when it + composed its own, is read live at ``/svc/``, and has no names when it serves no card (404 or no JSON). + Reading one fails otherwise, after ``CARD_ATTEMPTS`` tries, rather than route the site by names it didn't declare. + ``only`` keeps the websites of the envs it names, each of which must be deployed. Sites of different envs that + share a name are each named for their env (``_env_qualified``), so they're routed apart, while two that claim one + domain are still refused.""" + from agentenv_protocol import client as protocol_v1 + from agentenv_protocol.website import WEBSITE_BROWSE_URL_PARAM, WEBSITE_EXTENSION_URI + + from agent_env.env.env import gateway_url_of + + records = list(deployed_envs) + if only is not None: + missing = [env_id for env_id in only if env_id not in {d.env_id for d in deployed_envs}] + if missing: + raise ValueError(f"site_routing names {', '.join(missing)}, which this run hasn't deployed before the " + "computer") + records = [d for d in records if d.env_id in only] + found: list[tuple[str, dict]] = [] + for record in records: + address = gateway_url_of(record) or getattr(record, "environment_url", None) + frontends = getattr(record, "website_frontend_urls", None) or {} + named = set() + for child in protocol_v1.children(getattr(record, "environment_card", None) or {}): + if child.find_extension(WEBSITE_EXTENSION_URI) is None or not child.name: + continue + params = protocol_v1.extension_params(child.card, WEBSITE_EXTENSION_URI) + browse = params.get(WEBSITE_BROWSE_URL_PARAM) + service = _served_as(child) + url = (_joined(address, browse) if isinstance(browse, str) and browse and address + else frontends.get(service)) + if url is None: + logger.warning("site routing: the website %s of env %s names no route a browser reaches it at, so it " + "isn't routed", child.name, record.env_id) + continue + named.add(service) + found.append((record.env_id, _found(child.name, url, params))) + for name, url in frontends.items(): + if name not in named: + card = await _site_card(protocol_v1, f"{gateway_url_of(record)}/svc/{name}", name) + found.append((record.env_id, _found(name, url, protocol_v1.extension_params(card, WEBSITE_EXTENSION_URI)))) + return _env_qualified(found) + + +def _env_qualified(found: list[tuple[str, dict]]) -> list[dict]: + """Each site found, a name that sites of different envs share made ``-`` for each of them, letters, + digits, ``-`` and ``_`` kept, so every site of the run has a name of its own.""" + envs: dict[str, set[str]] = {} + for env_id, site in found: + envs.setdefault(site["name"], set()).add(env_id) + return [{**site, "name": re.sub(r"[^A-Za-z0-9_-]", "-", f"{env_id}-{site['name']}")} + if len(envs[site["name"]]) > 1 else site for env_id, site in found] + + +def _served_as(child) -> str: + """The gateway service a child env is served as: the ```` of the ``/svc//`` route its url is at, which + ``website_frontend_urls`` keys it by, else its card's name.""" + parts = urlsplit(child.url or "").path.strip("/").split("/") + return parts[1] if len(parts) > 1 and parts[0] == "svc" and parts[1] else child.name + + +def _found(name: str, url: str, params: dict) -> dict: + return {"name": name, "url": url, "domains": list(params.get("domains") or []), "live": bool(params.get("live")), + "allow_hosts": list(params.get("allow_hosts") or [])} + + +def _joined(address: str, url: str) -> str: + """``url`` resolved against ``address``: a full URL stands alone, and a path joins onto it.""" + return url if "://" in url else f"{address.rstrip('/')}/{url.lstrip('/')}" + + +async def _site_card(protocol_v1, url: str, name: str) -> dict: + """The card of the site ``name`` at ``url``, or none for a site that serves none.""" + for attempt in range(1, CARD_ATTEMPTS + 1): + try: + return await protocol_v1.get_card(url) + except httpx.HTTPStatusError as e: + if e.response.status_code == 404: + return {} + error: Exception = e + except ValueError: + return {} + except (httpx.HTTPError, OSError) as e: + error = e + if attempt < CARD_ATTEMPTS: + await asyncio.sleep(CARD_RETRY_S) + raise RuntimeError(f"site routing: the card of the site {name} at {url} couldn't be read, so its names are " + f"unknown: {type(error).__name__}: {error}") diff --git a/src/agent_env/env/envs/mcp_server.py b/src/agent_env/env/envs/mcp_server.py index 2f950f77a..96fad5ec0 100644 --- a/src/agent_env/env/envs/mcp_server.py +++ b/src/agent_env/env/envs/mcp_server.py @@ -107,13 +107,13 @@ def accept_toml(cls, data: dict, ctx: AuthoringContext) -> dict: @environment_card when it isn't set. Raises BundleError listing every problem.""" return ctx.accept_env(data, cls, named_by="image") - async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, cpu: float | None = None, memory_mb: int | None = None, sandbox_type: str | None = None, env_state_type: str | None = None, env_state_instance_id: str | None = None, artifact_id: str | None = None, artifact_version: int | None = None, *, attribution: Optional[Attribution] = None) -> DeployedEnv: + async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, cpu: float | None = None, memory_mb: int | None = None, sandbox_type: str | None = None, env_state_type: str | None = None, env_state_instance_id: str | None = None, artifact_id: str | None = None, artifact_version: int | None = None, *, attribution: Optional[Attribution] = None, keyed_roles: bool = False) -> DeployedEnv: # In container mode the server runs in its own container, so loads stage there, as for a MultiEnv child. return await deploy_through_provider( self, environment_name=self.environment_name, ttl_seconds=ttl_seconds, sandbox_type=sandbox_type, disk_size_gb=disk_size_gb, gateway_mode=gateway_mode, cpu=cpu, memory_mb=memory_mb, env_state_type=env_state_type, env_state_instance_id=env_state_instance_id, attribution=attribution, - artifact_id=artifact_id, artifact_version=artifact_version, + artifact_id=artifact_id, artifact_version=artifact_version, **({"keyed_roles": True} if keyed_roles else {}), ) def deploy_refusal(self, *, sandbox_type: str | None = None, **options) -> str | None: diff --git a/src/agent_env/env/envs/multi_env.py b/src/agent_env/env/envs/multi_env.py index e193771d2..3789e308b 100644 --- a/src/agent_env/env/envs/multi_env.py +++ b/src/agent_env/env/envs/multi_env.py @@ -91,6 +91,7 @@ def __init__(self, id: str, version: Optional[int], mcp_server_envs: list[MCPSer self._gateway_mode = GatewayMode.PERFORMANCE self._env_provider = None self._mcp_server_name: Optional[str] = None + self._role_key: Optional[str] = None self._deployed: Optional[DeployedEnv] = None self._replaced: list[tuple] = [] # the provider and sandbox of each deployment a later deploy replaced, for close() @@ -156,15 +157,17 @@ def accept_toml(cls, data: dict, ctx: AuthoringContext) -> dict: ctx.refuse(problems) return fields - async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, cpu: float | None = None, memory_mb: int | None = None, sandbox_type: str | None = None, env_state_type: str | None = None, env_state_instance_id: str | None = None, artifact_id: str | None = None, artifact_version: int | None = None, *, attribution: Optional[Attribution] = None) -> DeployedEnv: + async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, cpu: float | None = None, memory_mb: int | None = None, sandbox_type: str | None = None, env_state_type: str | None = None, env_state_instance_id: str | None = None, artifact_id: str | None = None, artifact_version: int | None = None, *, attribution: Optional[Attribution] = None, keyed_roles: bool = False, capture_traffic: bool = False) -> DeployedEnv: deployed_env = await deploy_through_provider( self, environment_name=None, ttl_seconds=ttl_seconds, sandbox_type=sandbox_type, disk_size_gb=disk_size_gb, gateway_mode=gateway_mode, cpu=cpu, memory_mb=memory_mb, env_state_type=env_state_type, env_state_instance_id=env_state_instance_id, attribution=attribution, - artifact_id=artifact_id, artifact_version=artifact_version, + artifact_id=artifact_id, artifact_version=artifact_version, **({"keyed_roles": True} if keyed_roles else {}), + **({"capture_traffic": True} if capture_traffic else {}), ) self._gateway_mode = gateway_mode self._mcp_server_name = deployed_env.mcp_server_name + self._role_key = deployed_env.role_key _hand_provider_to_child_envs(self) _hand_record_to_child_envs(self, deployed_env) return deployed_env @@ -190,6 +193,7 @@ async def from_deployed_env(cls, deployed: DeployedEnv) -> MultiEnv: _hand_record_to_child_envs(env, deployed) env._instance_id = deployed.instance_id env._mcp_server_name = deployed.mcp_server_name + env._role_key = deployed.role_key return env async def load_environment_artifact(self, environment_artifact: EnvironmentArtifact) -> None: diff --git a/src/agent_env/env/envs/playwright_mcp/entrypoint.sh b/src/agent_env/env/envs/playwright_mcp/entrypoint.sh index 337885849..68a602103 100644 --- a/src/agent_env/env/envs/playwright_mcp/entrypoint.sh +++ b/src/agent_env/env/envs/playwright_mcp/entrypoint.sh @@ -7,6 +7,8 @@ set -euo pipefail # The recorder serves the browser's card on RECORDER_PORT, restarted whenever it exits; the MCP server starts once # it listens, so the card is there by the time the server is healthy, or after RECORDER_WAIT_S without it. export RECORDER_PORT="${RECORDER_PORT:-18766}" +export PLAYWRIGHT_STORAGE_STATE="${PLAYWRIGHT_STORAGE_STATE:-/app/storage-state.json}" +[ -f "${PLAYWRIGHT_STORAGE_STATE}" ] || (umask 077 && echo '{"cookies": [], "origins": []}' > "${PLAYWRIGHT_STORAGE_STATE}") (while true; do /opt/recorder/bin/python /app/recorder.py || true; sleep 1; done) & for _ in $(seq 1 "$(( ${RECORDER_WAIT_S:-20} * 10 ))"); do (exec 3<>"/dev/tcp/127.0.0.1/${RECORDER_PORT}") 2>/dev/null && break @@ -17,6 +19,7 @@ exec npx @playwright/mcp@${PLAYWRIGHT_MCP_VERSION} \ --browser chromium \ --headless \ --isolated \ + --storage-state "${PLAYWRIGHT_STORAGE_STATE}" \ --no-sandbox \ --host "${MCP_HOST:-0.0.0.0}" \ --port "${MCP_PORT:-18765}" \ diff --git a/src/agent_env/env/envs/playwright_mcp/recorder-requirements.txt b/src/agent_env/env/envs/playwright_mcp/recorder-requirements.txt index 57ae4a2e3..4dc5eef92 100644 --- a/src/agent_env/env/envs/playwright_mcp/recorder-requirements.txt +++ b/src/agent_env/env/envs/playwright_mcp/recorder-requirements.txt @@ -1,4 +1,4 @@ -# The recorder installs the newest agentenv-framework-protocol at or above this floor, the first with computer-recording/v1; +# The recorder installs the newest agentenv-framework-protocol at or above this floor, the first with browser/v1's cookies; # raise it when the recorder needs a newer protocol. -agentenv-framework-protocol[computer]>=0.1.382 +agentenv-framework-protocol[computer]>=0.1.405 websockets==15.0.1 diff --git a/src/agent_env/env/envs/playwright_mcp/recorder.py b/src/agent_env/env/envs/playwright_mcp/recorder.py index c82dbedf2..e289d8a12 100644 --- a/src/agent_env/env/envs/playwright_mcp/recorder.py +++ b/src/agent_env/env/envs/playwright_mcp/recorder.py @@ -2,7 +2,12 @@ screencast frames, on a card and port of its own beside the MCP server. It is idle until a recording starts, which finds the browser through the remote debugging port Playwright opens for an ``--isolated`` Chromium and follows the page the agent is on. It runs standalone in the browser's image, so it imports only -``agentenv-framework-protocol[computer]`` and, to talk to the browser, ``websockets``.""" +``agentenv-framework-protocol[computer]`` and, to talk to the browser, ``websockets``. + +The same card serves browser/v1's ``cookies/set`` and ``cookies/clear``, so a step can start the browser signed in to +a site. Playwright MCP starts each browser context it opens from the storage state file ``--storage-state`` names +(``PLAYWRIGHT_STORAGE_STATE``), and launches its browser only at the agent's first tool call, so the cookies go into +that file, and through DevTools into every context of a browser already running.""" from __future__ import annotations import asyncio @@ -30,6 +35,10 @@ ) from agentenv_protocol.computer.card import COMPUTER_RECORDING from agentenv_protocol.computer.recording import RecordedFrame, RecordHints, Recording, Recordings, recording_params +from agentenv_protocol.errors import AgentEnvError +from agentenv_protocol.website.browser import (BROWSER_EXTENSION_URI, CLEAR_COOKIES, SET_COOKIES, Cookie, + devtools, domain_matches, storage_state) +from agentenv_protocol.website.browser import cookies as parse_cookies logger = logging.getLogger("playwright-mcp-recorder") @@ -43,6 +52,7 @@ #: How long one DevTools command may take. CALL_S = 10.0 JPEG_QUALITY = 80 +DEFAULT_STORAGE_STATE = "/app/storage-state.json" class CdpError(Exception): @@ -222,16 +232,106 @@ async def _show(self, cdp: Cdp, target: Optional[str]) -> None: self._session = session +class BrowserCookies: + """browser/v1's cookies: kept in the storage state file each new browser context starts from, and set in, or + cleared from, every context of the browser that is running, if one is. One change runs at a time, its file + write and DevTools calls together, so a concurrent set or clear never restores what another removed.""" + + def __init__(self, path: Path, find: Callable[[], Optional[int]], + connect: Callable[[int], AsyncContextManager[Cdp]]) -> None: + self.path, self._find, self._connect = path, find, connect + self._lock = asyncio.Lock() + + async def set(self, given: list[Cookie]) -> dict: + async with self._lock: + return await self._set(given) + + async def clear(self, domains: Optional[list[str]]) -> dict: + async with self._lock: + return await self._clear(domains) + + async def _set(self, given: list[Cookie]) -> dict: + kept = {c.key(): c for c in self._read()} + kept.update({c.key(): c for c in given}) + self._write(list(kept.values())) + + async def add(cdp: Cdp, context: dict) -> None: + await cdp.call("Storage.setCookies", {"cookies": devtools(given), **context}) + + return {"set": len(given), "open_browser": await self._each_context(add)} + + async def _clear(self, domains: Optional[list[str]]) -> dict: + before = self._read() + kept = [c for c in before if domains is not None and not c.matches(domains)] + self._write(kept) + + async def remove(cdp: Cdp, context: dict) -> None: + current = [] if domains is None else (await cdp.call("Storage.getCookies", context)).get("cookies") or [] + await cdp.call("Storage.clearCookies", context) + others = [_settable(c) for c in current if not domain_matches(c.get("domain", ""), domains or [])] + if others: + await cdp.call("Storage.setCookies", {"cookies": others, **context}) + + return {"cleared": len(before) - len(kept), "open_browser": await self._each_context(remove)} + + async def _each_context(self, act: Callable[[Cdp, dict], Any]) -> bool: + """``act`` on the default context and each other one of the running browser; False when none runs. An error + names the DevTools call's kind only, since its message may repeat a cookie.""" + port = await asyncio.to_thread(self._find) + if port is None: + return False + try: + async with self._connect(port) as cdp: + contexts = (await cdp.call("Target.getBrowserContexts")).get("browserContextIds") or [] + for context in [{}, *({"browserContextId": c} for c in contexts)]: + await act(cdp, context) + except (OSError, CdpError, httpx.HTTPError, KeyError, ValueError, asyncio.TimeoutError) as e: + raise AgentEnvError("browser_unreachable", f"the running browser didn't take the change " + f"({type(e).__name__}); the next context it opens starts with it", applied="unknown", + status=502) from None + return True + + def _read(self) -> list[Cookie]: + try: + state = json.loads(self.path.read_text(encoding="utf-8")) + except FileNotFoundError: + return [] + return [Cookie.model_validate({**c, "expires": None if c.get("expires", -1) in (-1, None) else c["expires"]}) + for c in state.get("cookies") or []] + + def _write(self, kept: list[Cookie]) -> None: + try: + origins = json.loads(self.path.read_text(encoding="utf-8")).get("origins") or [] + except (FileNotFoundError, ValueError): + origins = [] + staged = self.path.with_name(self.path.name + ".tmp") + staged.write_text(json.dumps({"cookies": storage_state(kept), "origins": origins}), encoding="utf-8") + os.chmod(staged, 0o600) + os.replace(staged, self.path) + + +def _settable(cookie: dict) -> dict: + """A cookie DevTools listed, as ``Storage.setCookies`` takes it back.""" + param = {k: cookie[k] for k in ("name", "value", "domain", "path", "secure", "httpOnly", "sameSite") if k in cookie} + if not cookie.get("session", False) and cookie.get("expires", -1) not in (-1, None): + param["expires"] = cookie["expires"] + return param + + @environment_card(name="playwright-mcp") class BrowserRecorder(AgentEnvEnvironment): - """computer-recording/v1 of the Playwright MCP browser's page; the browser's tools stay Playwright MCP's.""" + """computer-recording/v1 of the Playwright MCP browser's page, and browser/v1's cookies; the browser's tools stay + Playwright MCP's.""" recording = Recording() def __init__(self, find: Callable[[], Optional[int]] = find_browser, - connect: Callable[[int], AsyncContextManager[Cdp]] = connect_cdp) -> None: + connect: Callable[[int], AsyncContextManager[Cdp]] = connect_cdp, + storage_state_path: Optional[Path] = None) -> None: self._find, self._connect = find, connect self.recordings = Recordings(_Host(self)) + path = storage_state_path or Path(os.environ.get("PLAYWRIGHT_STORAGE_STATE", DEFAULT_STORAGE_STATE)) + self.cookies = BrowserCookies(path, find, connect) def extension_params(self, uri: str) -> dict: if uri == COMPUTER_RECORDING: @@ -250,6 +350,21 @@ async def _recording_start(self, request_id: str, scope: str = "display", window async def _recording_end(self, recording_id: str, upload: Optional[dict] = None, reason: str = "end") -> dict: return await self.recordings.end(recording_id, upload, reason) + @extension(BROWSER_EXTENSION_URI, op=SET_COOKIES, + description="The browser's cookies, for agentenv's steps, never its agent: cookies/set adds cookies " + "(name, value, domain, path, secure, httpOnly, expires, sameSite) to every context, open " + "and to come, and cookies/clear removes those of some domains, or all of them.") + async def _cookies_set(self, cookies: list[dict]) -> dict: + try: + given = parse_cookies(cookies) + except ValueError as e: + raise AgentEnvError("invalid_params", str(e)) from None + return await self.cookies.set(given) + + @extension(BROWSER_EXTENSION_URI, op=CLEAR_COOKIES) + async def _cookies_clear(self, domains: Optional[list[str]] = None) -> dict: + return await self.cookies.clear(domains) + async def frames(self, hints: RecordHints) -> AsyncIterator[RecordedFrame]: """The page at up to ``fps``: a frame each time it changes, and the last one again after ``HOLD_S``.""" screencast = Screencast(hints.max_edge, self._find, self._connect) diff --git a/src/agent_env/env/envs/website.py b/src/agent_env/env/envs/website.py index 64d4b2d5c..48848e21c 100644 --- a/src/agent_env/env/envs/website.py +++ b/src/agent_env/env/envs/website.py @@ -1,5 +1,11 @@ """Website environment: one site, served by one server image that holds its pages and API. A site -built before the server image held its pages may still name a separate frontend image, which is deprecated. +built before the server image held its pages may still name a separate frontend image, which is deprecated. Its +env.toml may carry the site's website spec, ``spec = "website.toml"`` or an inline ``[spec]`` table, whose site name, +health path and domains the env takes. + +Its ``settings`` are environment variables of the server's container, a live site's credentials among them: a setting +whose name reads as a secret's (a token, a password, ...) is stored as a ``secret:KEY`` or ``env:NAME`` reference, +never its value, and resolved from the deploying user's config only when the container is written. A deploy reads whether the site is ready from its env card, the protocol layer: ``website/v1``'s ``ready``, as ``computer-app/v1``'s says of an app. ``health_path`` (env.toml's ``health``) is read only for a site whose card has no @@ -15,9 +21,11 @@ import re import uuid from importlib.metadata import version as pkg_version +from pathlib import Path from typing import TYPE_CHECKING, Any, Callable, ClassVar, Optional from agentenv_protocol import client as protocol_v1 +from agentenv_protocol.website.spec import SpecError, WebsiteSpec, load as load_spec, resolve as resolve_spec from agent_env.artifact import Artifact, DockerImageArtifact, EnvironmentArtifact from agent_env.artifact.artifacts.docker_image import ProgressCallback, refuse_local_github_build from agent_env.entity.refs import EntityRef @@ -42,6 +50,9 @@ DEFAULT_HEALTH_PATH = AGENT_ENV_WEBSITE_HEALTH_PATH _HEALTH_PATH = re.compile(r"/[A-Za-z0-9._~/-]*") +_SETTING_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") +RESERVED_SETTINGS = frozenset({"DATABASE_URL", "SERVICE_NAME", "ENVIRONMENT_NAME", "PORT", "PYTHONUNBUFFERED", + "WEBSITE_ROLE_KEY"}) class WebsiteEnv(Env): @@ -51,7 +62,8 @@ class WebsiteEnv(Env): EntityRef.artifact("frontend_image", artifact_type="docker_image", optional=True)) toml_keys: ClassVar[dict[str, type]] = {"backend_image": object, "frontend_image": object, "environment_name": str, - "health": str, "env_provider_type": str} + "health": str, "env_provider_type": str, "spec": object, + "settings": dict} def __init__( self, @@ -65,6 +77,8 @@ def __init__( health_path: str = DEFAULT_HEALTH_PATH, metadata: Optional[dict[str, str]] = None, env_provider_type: str = "gateway", + domains: tuple[str, ...] | list[str] = (), + settings: Optional[dict[str, str]] = None, ): super().__init__(id, version, metadata=metadata) if not environment_name: @@ -79,6 +93,10 @@ def __init__( if problem := health_path_problem(health_path): raise ValueError(f"website env {id!r}: {problem}") self.health_path = health_path + self.domains = tuple(domains) + if problems := settings_problems(settings or {}): + raise ValueError(f"website env {id!r}: {'; '.join(problems)}") + self.settings = dict(settings or {}) self.environment_name = environment_name self.env_provider_type = env_provider_type self._sandbox = None @@ -106,9 +124,13 @@ def to_dict(self) -> dict: "type": self.frontend_docker_image_artifact.type, } base["health_path"] = self.health_path + if self.domains: + base["domains"] = list(self.domains) base["service_name"] = self.environment_name base["environment_name"] = self.environment_name # dual-write with service_name during the rename base["env_provider_type"] = self.env_provider_type + if self.settings: + base["settings"] = dict(self.settings) return base @classmethod @@ -125,6 +147,8 @@ def from_dict(cls, data: dict) -> WebsiteEnv: health_path=data.get("health_path", DEFAULT_HEALTH_PATH), metadata=data.get("metadata", {}), env_provider_type=data.get("env_provider_type", "gateway"), + domains=data.get("domains", ()), + settings=data.get("settings"), ) @classmethod @@ -147,7 +171,8 @@ def from_toml(cls, data: dict, ctx: AuthoringContext) -> WebsiteEnv: frontend_docker_image_artifact=ctx.artifact(frontend, DockerImageArtifact) if frontend else None, environment_name=fields["environment_name"], health_path=fields.get("health", DEFAULT_HEALTH_PATH), - env_provider_type=fields["env_provider_type"]) + env_provider_type=fields["env_provider_type"], + domains=fields.get("domains", ()), settings=fields.get("settings")) def to_toml(self, out: ExportContext) -> dict: """The env.toml of a folder this env's images are built from, which ``from_toml`` reads back as it.""" @@ -157,24 +182,41 @@ def to_toml(self, out: ExportContext) -> dict: toml["frontend_image"] = out.image(self.frontend_docker_image_artifact, "frontend_image") if self.health_path != DEFAULT_HEALTH_PATH: toml["health"] = self.health_path + if self.domains: + toml["spec"] = {"site": {"domains": list(self.domains)}} + if self.settings: + toml["settings"] = dict(self.settings) return {**toml, **self.provider_toml(out, self.env_provider_type)} @classmethod def accept_toml(cls, data: dict, ctx: AuthoringContext) -> dict: """The keys of ``data``, an env.toml, a website env takes, ``backend_image`` naming its server image, its - environment_name read from that image's @environment_card when it isn't set. Raises BundleError listing every - problem.""" + environment_name read from its spec's site name, else from that image's @environment_card, when it isn't set, + and its health path from the spec when it isn't set. Raises BundleError listing every problem.""" + spec, problems = website_spec_of(data, ctx.dir) if "spec" in data else (None, []) + if problems: + ctx.refuse([ctx.config_problem(p) for p in problems]) + if spec is not None: + data = {**({"environment_name": spec.site.name} if spec.site.name else {}), + **({"health": spec.site.health} if spec.site.health else {}), **data} health = data.get("health", DEFAULT_HEALTH_PATH) if isinstance(health, str) and (problem := health_path_problem(health)): ctx.refuse([ctx.config_problem(problem)]) - return ctx.accept_env(data, cls, named_by="backend_image") - - async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, cpu: float | None = None, memory_mb: int | None = None, sandbox_type: str | None = None, env_state_type: str | None = None, env_state_instance_id: str | None = None, artifact_id: str | None = None, artifact_version: int | None = None, *, attribution: Optional[Attribution] = None) -> DeployedEnv: + settings = data.get("settings") + if isinstance(settings, dict) and (problems := settings_problems(settings)): + ctx.refuse([ctx.config_problem(problem) for problem in problems]) + fields = ctx.accept_env(data, cls, named_by="backend_image") + if spec is not None and spec.site.domains: + fields["domains"] = spec.site.domains + return fields + + async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, cpu: float | None = None, memory_mb: int | None = None, sandbox_type: str | None = None, env_state_type: str | None = None, env_state_instance_id: str | None = None, artifact_id: str | None = None, artifact_version: int | None = None, *, attribution: Optional[Attribution] = None, keyed_roles: bool = False, capture_traffic: bool = False) -> DeployedEnv: return await deploy_through_provider( self, environment_name=self.environment_name, ttl_seconds=ttl_seconds, sandbox_type=sandbox_type, disk_size_gb=disk_size_gb, gateway_mode=gateway_mode, cpu=cpu, memory_mb=memory_mb, env_state_type=env_state_type, env_state_instance_id=env_state_instance_id, attribution=attribution, - artifact_id=artifact_id, artifact_version=artifact_version, + artifact_id=artifact_id, artifact_version=artifact_version, **({"keyed_roles": True} if keyed_roles else {}), + **({"capture_traffic": True} if capture_traffic else {}), ) def deploy_refusal(self, *, sandbox_type: str | None = None, **options) -> str | None: @@ -388,6 +430,53 @@ async def put_from_github( return env +def website_spec_of(data: dict, folder: Path) -> tuple[Optional[WebsiteSpec], list[str]]: + """The website spec an env.toml's ``spec`` names, a file relative to ``folder`` or an inline table, and the + problems with it: one that doesn't load, a live or replayed source, which a server image can't serve, and a site + name or health path that isn't the env.toml's own.""" + given = data.get("spec") + if given is None: + return None, [] + try: + if isinstance(given, str): + spec = load_spec(folder / given) + elif isinstance(given, dict): + spec = resolve_spec(given, folder, origin="spec") + else: + return None, [f"spec is the path of a website spec or a table of one, not {given!r}"] + except SpecError as e: + return None, [f"spec: {problem}" for problem in e.problems] + problems = [] + if spec.source is not None and spec.source.kind in ("live", "replay"): + problems.append(f"spec: a {spec.source.kind} source isn't served by a server image, so a website env can't " + "deploy it") + for key, ours in (("environment_name", spec.site.name), ("health", spec.site.health)): + if ours is not None and key in data and data[key] != ours: + problems.append(f"{key} {data[key]!r} isn't the spec's {ours!r}") + return spec, problems + + +def settings_problems(settings: dict) -> list[str]: + """Why ``settings`` can't be a website env's, each problem naming the setting and never its value: a name is an + environment variable's the gateway doesn't set itself, a value a string, and a secret's a bare ``secret:KEY`` or + ``env:NAME`` reference.""" + from agent_env.config.describe import is_secret_key + from agent_env.config.loader import ENV_REF_PREFIX, SECRET_REF_PREFIX + + problems = [] + for name, value in settings.items(): + if not isinstance(name, str) or not _SETTING_NAME.fullmatch(name) or name in RESERVED_SETTINGS: + problems.append(f"settings.{name}: a setting is an environment variable's name the gateway doesn't set " + f"itself ({', '.join(sorted(RESERVED_SETTINGS))} are its own)") + elif not isinstance(value, str): + problems.append(f"settings.{name} is a string, not a {type(value).__name__}") + elif is_secret_key(name) and not (value.startswith((SECRET_REF_PREFIX, ENV_REF_PREFIX)) and "?" not in value + and value not in (SECRET_REF_PREFIX, ENV_REF_PREFIX)): + problems.append(f"settings.{name} is a secret, so it takes a {SECRET_REF_PREFIX}KEY or {ENV_REF_PREFIX}NAME " + "reference with no ?default, not its value") + return problems + + def health_path_problem(health_path: str) -> str | None: """Why ``health_path`` can't be where a site answers once it's usable, or None: it's a path of plain characters, since the gateway writes it into a health check.""" diff --git a/src/agent_env/env/gateway/Dockerfile b/src/agent_env/env/gateway/Dockerfile index 73000f01d..4957dd39f 100644 --- a/src/agent_env/env/gateway/Dockerfile +++ b/src/agent_env/env/gateway/Dockerfile @@ -1,6 +1,6 @@ FROM public.ecr.aws/docker/library/python:3.11-slim@sha256:c20888b6acdd1e63e1c433a185bf3ad162c0288fe484616ce062e0d28add2900 -RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/* +RUN apt-get update && apt-get install -y curl tcpdump && rm -rf /var/lib/apt/lists/* WORKDIR /app diff --git a/src/agent_env/env/gateway/__init__.py b/src/agent_env/env/gateway/__init__.py index 6cd59de6f..cb94520d2 100644 --- a/src/agent_env/env/gateway/__init__.py +++ b/src/agent_env/env/gateway/__init__.py @@ -6,6 +6,7 @@ from .constants import ( AGENT_ENV_HIDDEN_BY_DEFAULT_META_KEY, AGENT_ENV_ROLE_HEADER, + AGENT_ENV_ROLE_KEY_HEADER, AGENT_ENV_ROLE_META_KEY, AGENT_ENV_SESSION_META_KEY, DEFAULT_ROLE, @@ -31,6 +32,7 @@ class InternalMCPServer: "AGENT_ENV_GATEWAY_MCP_PORT", "AGENT_ENV_HIDDEN_BY_DEFAULT_META_KEY", "AGENT_ENV_ROLE_HEADER", + "AGENT_ENV_ROLE_KEY_HEADER", "AGENT_ENV_ROLE_META_KEY", "AGENT_ENV_SESSION_META_KEY", "DEFAULT_ROLE", diff --git a/src/agent_env/env/gateway/caller.py b/src/agent_env/env/gateway/caller.py index 4c5c50edb..1546fe2c9 100644 --- a/src/agent_env/env/gateway/caller.py +++ b/src/agent_env/env/gateway/caller.py @@ -1,6 +1,7 @@ """Who the gateway acts for on one inbound request. -A ``Caller`` is the role the request's ``AgentEnv-Role`` header named (``default`` when it named none) and the key of +A ``Caller`` is the role the request's ``AgentEnv-Role`` header named (``default`` when it named none, or named another +without the deploy's ``AgentEnv-Role-Key`` when the deploy has one) and the key of the MCP session the request belongs to (empty on the REST and harness paths, which have no session). It is bound in three scopes, which read in order in ``gateway.py``: per HTTP request by the served app (the role), per inbound MCP session by the session lifespan (the key, kept for every tools/call of that session), and never per process. Every diff --git a/src/agent_env/env/gateway/clock.py b/src/agent_env/env/gateway/clock.py index cceb14364..dda1dd7eb 100644 --- a/src/agent_env/env/gateway/clock.py +++ b/src/agent_env/env/gateway/clock.py @@ -217,11 +217,13 @@ def at_elapsed(self, elapsed: float) -> datetime | None: return _advance(self._t0, elapsed - self._lived + self._ahead) def read(self) -> dict: - """The frozen consumer contract: {"virtual_time": ""} — pure read, never advances.""" - current = self.now() - if current is None: - raise ClockError("clock is not armed") - return {"virtual_time": _iso(current)} + """The frozen consumer contract: {"virtual_time": ""} — pure read, never advances — plus the rate it + runs at, "virtual_seconds_per_real_second", so a reader extrapolates between reads without guessing it.""" + with self._lock: + if not self._armed or self._t0 is None: + raise ClockError("clock is not armed") + current = _advance(self._t0, (self._now() - self._anchor) * self._rate + self._ahead) + return {"virtual_time": _iso(current), "virtual_seconds_per_real_second": self._rate} def state(self) -> dict: with self._lock: diff --git a/src/agent_env/env/gateway/constants.py b/src/agent_env/env/gateway/constants.py index 30bd00031..03e910a3c 100644 --- a/src/agent_env/env/gateway/constants.py +++ b/src/agent_env/env/gateway/constants.py @@ -81,6 +81,12 @@ def data_plane_load_timeout_s(size_bytes: int | None) -> int: EXT_NOTIFICATIONS_URI = "urn:agentenv:notifications/v1" EXT_WEBSITE_URI = "urn:agentenv:website/v1" WEBSITE_READY_OP = "ready" +EXT_TRAFFIC_URI = "urn:agentenv:traffic/v1" +TRAFFIC_LIST_OP = "list" +TRAFFIC_GET_OP = "get" +TRAFFIC_CAPTURE_DIR_ENV = "TRAFFIC_CAPTURE_DIR" +PCAP_MEDIA_TYPE = "application/vnd.tcpdump.pcap" +WEBSITE_BROWSE_URL_PARAM = "browse_url" # Rows per GET /changes page: the default, and the most a client may ask for. CHANGES_PAGE_DEFAULT = 1000 @@ -126,9 +132,10 @@ def sources(names: list[str]) -> str: TRIGGER_EXECUTOR_HOURS_TIME_ZONE = "executor_hours_time_zone" TRIGGER_NEXT_EVENT_AT = "next_event_at" TRIGGER_NOTIFY = "notify" +TRIGGER_WEBSITE_OP = "website_op" TRIGGER_FEATURES = (TRIGGER_NAMED_EXECUTORS, TRIGGER_PROVOKING_ROLES, TRIGGER_ACTION_DELAY, TRIGGER_EXECUTOR_HOURS, TRIGGER_UNLESS, TRIGGER_WEBSITE, TRIGGER_EXECUTOR_HOURS_TIME_ZONE, TRIGGER_NEXT_EVENT_AT, - TRIGGER_NOTIFY) + TRIGGER_NOTIFY, TRIGGER_WEBSITE_OP) # The one `source` a /step call may name instead of step_api, with the id of the hook artifact that made it. STEP_SOURCE_POST_TURN_HOOK = "post_turn_hook" @@ -142,6 +149,13 @@ def sources(names: list[str]) -> str: ) AGENT_ENV_ROLE_HEADER = "AgentEnv-Role" +AGENT_ENV_ROLE_KEY_HEADER = "AgentEnv-Role-Key" +# Sent by a computer's site proxy, which serves a website's route at the root of the site's own name. +AGENT_ENV_SITE_ROOT_HEADER = "AgentEnv-Site-Root" +GATEWAY_ROLE_KEY_ENV = "AGENTENV_ROLE_KEY" +GATEWAY_PROVIDER_HEADERS_ENV = "AGENTENV_PROVIDER_REQUEST_HEADERS" +CONTROL_TOKEN_ENV = "AGENTENV_CONTROL_TOKEN" +WEBSITE_ROLE_KEY_ENV = "WEBSITE_ROLE_KEY" # MCP request `_meta` keys the gateway stamps on every child tools/call: the caller's role, and on the MCP path a # key naming the inbound session, so a child can keep per-caller state behind the gateway's one shared session. AGENT_ENV_ROLE_META_KEY = "agentenv.io/role" @@ -264,3 +278,12 @@ def random_mcp_server_name() -> str: "required": False, }, ] +TRAFFIC_EXTENSION = { + "uri": EXT_TRAFFIC_URI, + "description": "Traffic (urn:agentenv:traffic/v1): the packets each website's server and frontend containers sent and received, captured by tcpdump in each container's network namespace, from just after the container starts, as rotating classic pcap files. list names each capture file with the container it holds, its size and when it was last written, oldest first; get streams one as application/vnd.tcpdump.pcap. A container's oldest file is overwritten once its rotation is full. Harness control path: with a control token, each call takes it as its bearer.", + "params": {"endpoint": "/traffic/captures", "format": "pcap", "methods": { + TRAFFIC_LIST_OP: {"method": "GET", "endpoint": "/traffic/captures", "request": {"type": "object", "properties": {}}}, + TRAFFIC_GET_OP: {"method": "GET", "endpoint": "/traffic/capture", "request": {"type": "object", "properties": { + "name": {"type": "string", "minLength": 1}}, "required": ["name"]}}}}, + "required": False, +} diff --git a/src/agent_env/env/gateway/entrypoint.py b/src/agent_env/env/gateway/entrypoint.py index b0b936ae9..451f718e5 100644 --- a/src/agent_env/env/gateway/entrypoint.py +++ b/src/agent_env/env/gateway/entrypoint.py @@ -7,7 +7,8 @@ import httpx from . import AGENT_ENV_GATEWAY_MCP_PORT, GatewayMode, InternalMCPServer -from .constants import DEFAULT_MCP_SERVER_NAME +from .constants import (CONTROL_TOKEN_ENV, DEFAULT_MCP_SERVER_NAME, GATEWAY_PROVIDER_HEADERS_ENV, GATEWAY_ROLE_KEY_ENV, + TRAFFIC_CAPTURE_DIR_ENV) from .gateway import Gateway logger = logging.getLogger(__name__) @@ -152,5 +153,9 @@ def parse_name_url_map(env_str: str) -> dict[str, str]: rest_proxy_urls=rest_proxy_urls, gateway_mode=gateway_mode, service_db_url=service_db_url, + role_key=os.environ.get(GATEWAY_ROLE_KEY_ENV) or None, + provider_headers=os.environ.get(GATEWAY_PROVIDER_HEADERS_ENV, "").split(","), + traffic_dir=os.environ.get(TRAFFIC_CAPTURE_DIR_ENV) or None, + control_token=os.environ.get(CONTROL_TOKEN_ENV) or None, ) gateway.run() diff --git a/src/agent_env/env/gateway/gateway.py b/src/agent_env/env/gateway/gateway.py index 73f6358be..427b6320e 100644 --- a/src/agent_env/env/gateway/gateway.py +++ b/src/agent_env/env/gateway/gateway.py @@ -4,12 +4,16 @@ import asyncio import atexit import contextlib +import hmac import json import logging import os import posixpath +import re import socket import subprocess +import tempfile +from collections.abc import Iterable from contextlib import asynccontextmanager from datetime import datetime, timezone from pathlib import Path @@ -30,8 +34,10 @@ from mcp.shared.exceptions import McpError from mcp.types import CONNECTION_CLOSED, CallToolResult, ErrorData, TextContent, Tool as MCPTool from pydantic import create_model -from starlette.requests import Request +from starlette.requests import HTTPConnection, Request from starlette.responses import Response, StreamingResponse +from starlette.routing import WebSocketRoute +from starlette.websockets import WebSocket from anyio import ClosedResourceError if TYPE_CHECKING: @@ -41,6 +47,7 @@ from .constants import ( AGENT_ENV_HIDDEN_BY_DEFAULT_META_KEY, AGENT_ENV_ROLE_HEADER, + AGENT_ENV_ROLE_KEY_HEADER, AGENT_ENV_ROLE_META_KEY, # noqa: F401 (the server module serves the whole vocabulary, used in this file or not) AGENT_ENV_SESSION_META_KEY, # noqa: F401 CARD_FETCH_TIMEOUT_S, @@ -49,6 +56,7 @@ CHILD_DISCOVERY_TIMEOUT_S, DATA_PLANE_LOAD_TIMEOUT_MAX_S, DEFAULT_ROLE, + EXT_WEBSITE_URI, GATEWAY_COMPOSE_HOST, GATEWAY_EXTENSIONS, GATEWAY_TRAJECTORY_FILE, @@ -56,12 +64,15 @@ METHOD_ADD, METHOD_GET, METHOD_RESET, + PCAP_MEDIA_TYPE, PROTOCOL_VERSION, RPC_PATH, STATE_TOOL_NAME_CLASHES_KEY, STEP_SOURCE_POST_TURN_HOOK, TOOL_DISABLE_ACTION, TOOL_ENABLE_ACTION, + TRAFFIC_EXTENSION, + WEBSITE_BROWSE_URL_PARAM, WELL_KNOWN_PATH, WILDCARD, GatewayMode, @@ -72,7 +83,21 @@ from .child_transport import child_http_client from .clock import Clock, ClockError, _iso from .get_time import GET_TIME_TOOL_NAME, build_get_time_tool -from .rest_proxy import ROLE_HEADER_BYTES, UpstreamResponse, fetch_child_card, forward, rewrite_child_card, upstream_url +from .rest_proxy import ( + ROLE_HEADER_BYTES, + ROLE_KEY_HEADER_BYTES, + UpstreamResponse, + WebsiteRoute, + fetch_child_card, + forward, + operations, + refuse_websocket, + relay_websocket, + rewrite_child_card, + route_name, + upstream_url, + website_routes, +) from .triggers import TriggerEngine, TriggerError from .notifications import CHECK_TOOL_NAMES, NotificationError, Notifications, build_check_tools @@ -81,6 +106,9 @@ # RFC 9110's safe methods: a request using one asks for no change, so the proxy measures no write window for it. _SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS", "TRACE"}) _STEP_API = {"source": "step_api"} +_CAPTURE_FILE = re.compile(r"(?P[A-Za-z0-9][A-Za-z0-9_.-]*)\.pcap(?P[0-9]*)") +_CONTROL_SCOPE_KEY = "agentenv.control" +_AUTHORIZATION = b"authorization" class ToolNameClash(ValueError): @@ -108,6 +136,8 @@ class Gateway: REST_PROXY_TIMEOUT_S = float(DATA_PLANE_LOAD_TIMEOUT_MAX_S) # data-plane loads REST_PROXY_CONNECT_TIMEOUT_S = 10.0 # a service that never answers the TCP connect fails in seconds, not an hour REST_PROXY_TIMEOUT = httpx.Timeout(REST_PROXY_TIMEOUT_S, connect=REST_PROXY_CONNECT_TIMEOUT_S) + CAPTURE_READ_BYTES = 64 * 1024 + CAPTURE_SNAPSHOT_ATTEMPTS = 3 def __init__( self, @@ -119,15 +149,35 @@ def __init__( rest_proxy_urls: dict[str, str] | None = None, gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, service_db_url: str | None = None, + role_key: str | None = None, + provider_headers: Iterable[str] = (), + traffic_dir: str | None = None, + control_token: str | None = None, ): + """`role_key` is the deploy's secret: with one, a request is believed to act as a role other than `default` + only when it carries the key beside the role; without one, as before it, the role header alone is believed. + `provider_headers` names the headers the sandbox provider in front of the gateway is sent, its credentials, + which the gateway takes off every request it forwards to a service behind it. + + `traffic_dir` is the folder the deploy's capture sidecars write their websites' pcap files to, mounted here + read-only; with one, the gateway serves traffic/v1 from it. `control_token` is the bearer its framework-only + routes take (traffic/v1's): a request carrying it has it taken off before anything else sees it, so no site or + backing server it is proxied to ever reads it; without one, those routes are open, as an env's are without + ``AGENTENV_CONTROL_TOKEN``.""" self.host = host self.port = port self.server_name = server_name self.internal_mcp_servers = internal_mcp_servers self.website_urls = website_urls or {} self.rest_proxy_urls = rest_proxy_urls or {} + self._site_reserved: dict[str, frozenset[str]] = {} self.gateway_mode = gateway_mode self._service_db_url = service_db_url + self._role_key = role_key or None + self._withheld = frozenset(name.strip().lower().encode() for name in provider_headers if name.strip()) + self._traffic_dir = Path(traffic_dir) if traffic_dir else None + self._control_bearer = f"Bearer {control_token}".encode() if control_token else None + self._website_routes: dict[str, list[WebsiteRoute] | None] = {} self._db_conn: psycopg2.extensions.connection | None = None if service_db_url: self._db_conn = psycopg2.connect(service_db_url) @@ -190,18 +240,23 @@ async def proxy_to_service(request: Request) -> Response: if self.website_urls: @self._mcp.custom_route( "/website/{service_name}/{path:path}", - methods=["GET", "POST", "PUT", "DELETE", "PATCH"], + methods=["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"], ) async def proxy_to_frontend(request: Request) -> Response: return await self._proxy_website_request(request) @self._mcp.custom_route( "/website/{service_name}", - methods=["GET", "POST", "PUT", "DELETE", "PATCH"], + methods=["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"], ) async def proxy_to_frontend_root(request: Request) -> Response: return await self._proxy_website_request(request) + self._mcp._custom_starlette_routes += [ + WebSocketRoute("/website/{service_name}/{path:path}", self._proxy_website_socket), + WebSocketRoute("/website/{service_name}", self._proxy_website_socket), + ] + @self._mcp.custom_route(f"/tools/{TOOL_DISABLE_ACTION}", methods=["POST"]) async def tools_disable(request: Request) -> Response: return await self._handle_role_mutation(request, action=TOOL_DISABLE_ACTION) @@ -239,13 +294,15 @@ async def get_state(request: Request) -> Response: async def step_endpoint(request: Request) -> Response: return await self._handle_step(request) - async def _trigger_write(request: Request, method) -> Response: - """Shared body for the trigger write routes: parse JSON, invoke the engine method, - map a TriggerError (bad config) to a 400.""" + async def _trigger_write(request: Request, method, prepare=None) -> Response: + """Shared body for the trigger write routes: parse JSON, await `prepare` with it, invoke the engine + method, map a TriggerError (bad config) to a 400.""" try: body = await request.json() except Exception: return self._json_response({"ok": False, "error": "Invalid JSON body"}, 400) + if prepare is not None: + await prepare(body) try: return self._json_response(method(body), 200) except TriggerError as e: @@ -253,7 +310,7 @@ async def _trigger_write(request: Request, method) -> Response: @self._mcp.custom_route("/triggers/register", methods=["POST"]) async def triggers_register(request: Request) -> Response: - return await _trigger_write(request, self._trigger_engine.register) + return await _trigger_write(request, self._trigger_engine.register, prepare=self._read_trigger_website_routes) @self._mcp.custom_route("/triggers/remove", methods=["POST"]) async def triggers_remove(request: Request) -> Response: @@ -376,6 +433,16 @@ async def notifications_clear(request: Request) -> Response: async def env_trajectory(request: Request) -> Response: return self._serve_trajectory() + if self._traffic_dir is not None: + @self._mcp.custom_route("/traffic/captures", methods=["GET"]) + async def traffic_captures(request: Request) -> Response: + return self._refuse_uncontrolled(request) or await asyncio.to_thread(self._serve_captures) + + @self._mcp.custom_route("/traffic/capture", methods=["GET"]) + async def traffic_capture(request: Request) -> Response: + return self._refuse_uncontrolled(request) or await asyncio.to_thread( + self._serve_capture, request.query_params.get("name", "")) + @self._mcp.custom_route("/changes", methods=["GET"]) async def env_changes(request: Request) -> Response: return await self._serve_changes(request) @@ -407,7 +474,7 @@ def run(self) -> None: anyio.run(server.serve) # The three scopes a Caller is bound in, innermost first. Request: the served app binds the role the - # AgentEnv-Role header named. Session: mcp's Server.run enters the session lifespan once per inbound MCP + # AgentEnv-Role header named, as `_attributed_role` believes it. Session: mcp's Server.run enters the session lifespan once per inbound MCP # session, in the session's server task before any handler task is spawned, so the key bound there is inherited # by every tools/call of that session (and the role is the one its initialize request carried, which that task # inherited). Process: child sessions and the time-trigger driver, outer to the MCP session manager's lifespan. @@ -419,14 +486,18 @@ def _asgi_app(self): inner.router.lifespan_context = self._process_lifespan(inner.router.lifespan_context) async def app(scope, receive, send): + if scope["type"] in ("http", "websocket") and self._control_bearer is not None: + scope = self._take_control_token(scope) if scope["type"] != "http": await inner(scope, receive, send) return - role = DEFAULT_ROLE + claimed, given_key = None, None for name, value in scope.get("headers", ()): - if name == ROLE_HEADER_BYTES: - role = value.decode(errors="ignore").strip() or DEFAULT_ROLE - break + if name == ROLE_HEADER_BYTES and claimed is None: + claimed = value.decode(errors="ignore") + elif name == ROLE_KEY_HEADER_BYTES and given_key is None: + given_key = value.decode(errors="ignore") + role = self._attributed_role(claimed or "", given_key or "") token = Caller(role=role, session=Caller.current().session).bind() try: await inner(scope, receive, send) @@ -435,6 +506,22 @@ async def app(scope, receive, send): return app + def _take_control_token(self, scope: dict) -> dict: + """``scope`` without any Authorization header that carries the gateway's control token, and marked when one + did, so a framework-only route can tell and nothing the request is proxied to ever reads the token.""" + headers = scope.get("headers", ()) + kept = [(name, value) for name, value in headers + if not (name == _AUTHORIZATION and hmac.compare_digest(value, self._control_bearer))] + return {**scope, "headers": kept, _CONTROL_SCOPE_KEY: len(kept) != len(headers)} + + def _refuse_uncontrolled(self, request: Request) -> Response | None: + """The refusal of a framework-only route's request that didn't carry the control token, or None: with no token, + every request is let through.""" + if self._control_bearer is None or request.scope.get(_CONTROL_SCOPE_KEY): + return None + return self._json_response({"ok": False, "error": {"code": "unauthorized", "message": + "this route is agent-env's own: it takes the env's control token"}}, 401) + @asynccontextmanager async def _session_lifespan(self, mcp: FastMCP): """Once per inbound MCP session: the session's key joins the caller, and a connect gets to heal a failed @@ -602,7 +689,8 @@ async def _proxy_rest_request(self, request: Request, url_map: dict[str, str] | role = self._request_role(request) mcp_server = url_map is self.rest_proxy_urls and service_name.startswith("mcp-") if request.method in _SAFE_METHODS: - return await forward(request, base_url, role=role, timeout=self.REST_PROXY_TIMEOUT, mcp_server=mcp_server) + return await forward(request, base_url, role=role, timeout=self.REST_PROXY_TIMEOUT, mcp_server=mcp_server, + role_key=self._role_key, withheld=self._withheld) changelog_before = self._query_changelog_id() attribution = self._attribution(role) # at the start: a fire may be gone by the time the response ends @@ -610,12 +698,16 @@ async def log_write(status: int) -> None: changelog_after = self._query_changelog_id() if changelog_after is None or changelog_after <= (changelog_before or 0): return # only a request that wrote changelog rows is a write window + path = "/" + request.path_params.get("path", "") + op = (self._website_op(service_name, request.method, site_request_path(request) or path) + if url_map is self.website_urls else None) await self._log_event({ "event_type": "http_request", "route": "website" if url_map is self.website_urls else "svc", "service": service_name, "method": request.method, - "path": "/" + request.path_params.get("path", ""), + "path": path, + **({"op": op} if op else {}), "status": status, **attribution, "changelog_id_before": changelog_before, @@ -623,21 +715,26 @@ async def log_write(status: int) -> None: }) return await forward(request, base_url, role=role, timeout=self.REST_PROXY_TIMEOUT, mcp_server=mcp_server, - on_done=log_write) + role_key=self._role_key, on_done=log_write, withheld=self._withheld) async def _proxy_website_request(self, request: Request) -> Response: """Proxy a website request; with a website or state trigger registered, feed it to the engine and hold barriers. - A site's data plane and card are never reached through its route, so an agent browsing the site can't read - what grades it; agentenv reaches them through /svc//.""" - if reserved_site_request(request): + A site's data plane and card, and every path its card advertises for agentenv (``site_reserved_paths``), are + never reached through its route, so an agent browsing the site can't read its data or what grades it; agentenv + reaches them through /svc//.""" + reserved = await self._site_reserved_paths(request.path_params["service_name"]) + if reserved is None: + return self._json_response({"error": "website card unavailable"}, 503) + if reserved_site_request(request, reserved): return self._json_response({"error": "not found"}, 404) response = await self._proxy_rest_request(request, url_map=self.website_urls) app = request.path_params["service_name"] engine = self._trigger_engine if app in self.website_urls and (engine.has_website_triggers or engine.has_state_triggers): + path = site_request_path(request) or "/" + request.path_params.get("path", "") if isinstance(response, UpstreamResponse): - if not engine.may_hold_website_request(self._request_role(request), app, request.method, - "/" + request.path_params.get("path", ""), response.status_code): + if not engine.may_hold_website_request(self._request_role(request), app, request.method, path, + response.status_code, self._website_op(app, request.method, path)): response.after_body(lambda status: self._fire_website_triggers(request, app, status)) # answered at its last byte return response await response.read_ahead_if_sized(self.BARRIER_READ_AHEAD_MAX_BYTES) # its window ends before its triggers run @@ -645,22 +742,106 @@ async def _proxy_website_request(self, request: Request) -> Response: await self._await_barriers(barrier_tasks, barrier_timeout_s) return response + async def _proxy_website_socket(self, websocket: WebSocket) -> None: + """Relay a WebSocket on a site's route to its server, as its HTTP requests are: its data plane and card, and + every path its card advertises for agentenv (``site_reserved_paths``), are refused, and it goes upstream with + the role, role key and forwarded address an HTTP request would carry.""" + reserved = await self._site_reserved_paths(websocket.path_params["service_name"]) + if reserved is None: + await refuse_websocket(websocket, self._json_response({"error": "website card unavailable"}, 503)) + return + if reserved_site_request(websocket, reserved): + await refuse_websocket(websocket, self._json_response({"error": "not found"}, 404)) + return + service_name = websocket.path_params["service_name"] + base_url = self.website_urls.get(service_name) + if base_url is None: + await refuse_websocket(websocket, self._json_response({"error": f"Unknown service: {service_name}"}, 404)) + return + await relay_websocket(websocket, base_url, role=self._request_role(websocket), role_key=self._role_key, + open_timeout=self.REST_PROXY_CONNECT_TIMEOUT_S, withheld=self._withheld) + async def _fire_website_triggers(self, request: Request, app: str, status: int) -> tuple[list[asyncio.Task], float | None]: """Feed an answered website request to the trigger engine (fail-open).""" try: # Only a website trigger reads the request's args; a state re-check needs just the method and status. args = _request_args(request, await request.body()) if self._trigger_engine.has_website_triggers else {} - return self._trigger_engine.on_website_request(self._request_role(request), app, request.method, - "/" + request.path_params.get("path", ""), status, args) + path = site_request_path(request) or "/" + request.path_params.get("path", "") + return self._trigger_engine.on_website_request(self._request_role(request), app, request.method, path, + status, args, self._website_op(app, request.method, path)) except Exception: logger.exception("Trigger evaluation failed (fail-open)") return [], None - @staticmethod - def _request_role(request: Request) -> str: - """The role the gateway attributes a REST request to: the first AgentEnv-Role header, else the default role.""" - return request.headers.get(AGENT_ENV_ROLE_HEADER, "").strip() or DEFAULT_ROLE + def _request_role(self, request: HTTPConnection) -> str: + """The role the gateway attributes a REST request to: the first AgentEnv-Role header, as `_attributed_role` + believes it, else the default role.""" + return self._attributed_role(request.headers.get(AGENT_ENV_ROLE_HEADER, ""), + request.headers.get(AGENT_ENV_ROLE_KEY_HEADER, "")) + + def _attributed_role(self, claimed: str, given_key: str) -> str: + """The role a request claiming `claimed` acts as: `default` when it claims none, and when the deploy has a + role key the request did not send. Without a key (a deploy without keyed_roles) the claim alone is believed.""" + role = claimed.strip() or DEFAULT_ROLE + if self._role_key is None or role == DEFAULT_ROLE: + return role + return role if hmac.compare_digest(given_key.encode(), self._role_key.encode()) else DEFAULT_ROLE + + async def _read_trigger_website_routes(self, body: Any) -> None: + """Read the routes of each website a website trigger in a register `body` names an `op` of, so registration + can check the op and requests can be matched by it; one whose card could not be read before is read again.""" + triggers = body.get("triggers") if isinstance(body, dict) else None + apps = {trigger["when"]["app"] for trigger in triggers if isinstance(trigger, dict) + and isinstance(trigger.get("when"), dict) and trigger["when"].get("type") == "website" + and trigger["when"].get("op") is not None and isinstance(trigger["when"].get("app"), str) + and trigger["when"]["app"] in self.website_urls} if isinstance(triggers, list) else set() + await asyncio.gather(*(self._read_website_routes(app, again=True) for app in apps)) + + async def _read_website_routes(self, app: str, *, again: bool = False) -> None: + """Read website `app`'s routes from its card once (`again`: once more, when its card could not be read). The + env card's composition reads them too, so a deploy, which reads that card, leaves them read.""" + if app in self._website_routes and not (again and self._website_routes[app] is None): + return + async with httpx.AsyncClient() as client: + card = await fetch_child_card(client, app, self.rest_proxy_urls.get(app) or self.website_urls[app]) + self._website_routes[app] = website_routes(card) if card is not None else None + + def website_route_names(self, app: str) -> set[str] | None: + """The names of the routes website `app`'s card names, as last read; None when it has not been read.""" + routes = self._website_routes.get(app) + return None if routes is None else {route.name for route in routes if route.name is not None} + + def _website_op(self, app: str, method: str, path: str) -> str | None: + """The name of the route of website `app` a request reached, by the routes last read from its card; never a + read of its own, so answering a request waits on no card.""" + return route_name(self._website_routes.get(app) or [], method, path) + + async def _site_reserved_paths(self, app: str) -> frozenset[str] | None: + """The paths ``app``'s card advertises for agentenv (``site_reserved_paths``), read once from its server, the one + its /svc/ route reaches (else the one its website route does), and kept; none for a site serving no card, one + whose card path answers a 4xx or what isn't a JSON object, and None, read again next time, while its card can't + be read, unreachable or answering a 5xx, so its route serves nothing until it knows what to refuse.""" + if app in self._site_reserved: + return self._site_reserved[app] + base_url = self.rest_proxy_urls.get(app) or self.website_urls.get(app) + if base_url is None: + return frozenset() + try: + async with httpx.AsyncClient() as client: + response = await client.get(f"{base_url.rstrip('/')}{WELL_KNOWN_PATH}", timeout=CARD_FETCH_TIMEOUT_S) + if response.status_code >= 500: + response.raise_for_status() + except Exception as e: + logger.warning(f"website {app}: its card could not be read ({type(e).__name__}: {e}), so its route answers " + "503 until it can be") + return None + try: + card = response.json() if response.is_success else None + except ValueError: + card = None + self._site_reserved[app] = site_reserved_paths(card) if isinstance(card, dict) else frozenset() + return self._site_reserved[app] def _backing_keys(self) -> list[str]: # rest_proxy_urls entries are the env's backing servers: mcp- (MCP) + (website @@ -668,15 +849,26 @@ def _backing_keys(self) -> list[str]: return list(self.rest_proxy_urls) def _rewrite_child_card(self, key: str, card: dict) -> dict: - return rewrite_child_card(key, card, self._gateway_extensions()) + """One backing server's card rooted at the gateway (``rewrite_child_card``); a website's ``website/v1`` params + also name the route the gateway serves it to a browser at, ``/website//``, as ``browse_url``.""" + card = rewrite_child_card(key, card, self._gateway_extensions()) + if key in self.website_urls: + for ext in (card.get("capabilities") or {}).get("extensions") or []: + if isinstance(ext, dict) and ext.get("uri") == EXT_WEBSITE_URI: + params = ext.get("params") if isinstance(ext.get("params"), dict) else {} + ext["params"] = {**params, WEBSITE_BROWSE_URL_PARAM: f"/website/{key}/"} + return card async def _fetch_child_card(self, client: httpx.AsyncClient, key: str, base_url: str) -> dict | None: - """Fetch + rewrite one backing server's card; None (omit) on 404 / error / timeout / malformed JSON.""" + """Fetch + rewrite one backing server's card; None (omit) on 404 / error / timeout / malformed JSON. A + website's card read here also refreshes the routes its requests are named by.""" child = await fetch_child_card(client, key, base_url) + if child is not None and key in self.website_urls: + self._website_routes[key] = website_routes(child) return None if child is None else self._rewrite_child_card(key, child) def _gateway_extensions(self) -> list[dict]: - return GATEWAY_EXTENSIONS + return [*GATEWAY_EXTENSIONS, TRAFFIC_EXTENSION] if self._traffic_dir is not None else GATEWAY_EXTENSIONS async def _serve_env_card(self) -> Response: """Compose the env card: the gateway's own extensions + each backing server's card nested @@ -1421,6 +1613,84 @@ def _read_changelog(self, after_id: int, count: int) -> list[dict]: row["timestamp"] = row["timestamp"].isoformat() return rows + def _capture_files(self) -> list[tuple[Path, re.Match]]: + """Each capture file in the traffic folder, as tcpdump's rotation names it, with its name's parts; a link is + none, so nothing but what the captures wrote is ever served.""" + try: + entries = list(self._traffic_dir.iterdir()) + except FileNotFoundError: + return [] + return [(path, match) for path in entries + if (match := _CAPTURE_FILE.fullmatch(path.name)) and path.is_file() and not path.is_symlink()] + + def _serve_captures(self) -> Response: + """traffic/v1's list: each capture file's name, the container it holds, its size and when it was last written, + oldest first, so a container's files read in the order their packets were captured.""" + found = [] + for path, match in self._capture_files(): + try: + stat = path.stat() + except FileNotFoundError: + continue + found.append(((match["container"], stat.st_mtime_ns, path.name), { + "name": path.name, "container": match["container"], "bytes": stat.st_size, + "modified_at": datetime.fromtimestamp(stat.st_mtime, timezone.utc).isoformat()})) + captures = [capture for _, capture in sorted(found, key=lambda entry: entry[0])] + return self._json_response({"format": "pcap", "captures": captures}, 200) + + def _serve_capture(self, name: str) -> Response: + """traffic/v1's get: the capture file ``name``, one ``list`` names, as a copy taken whole before any of it is + sent (``_snapshot_capture``), so a file tcpdump's rotation rewrites while it downloads never arrives mixed or + cut short; 503 when every copy was rewritten under it. A file still being written ends on a whole packet, or at + worst one torn.""" + if not _CAPTURE_FILE.fullmatch(name): + return self._json_response({"ok": False, "error": {"code": "invalid_params", + "message": f"{name!r} is not a capture's name, as list gives it"}}, 400) + path = self._traffic_dir / name + try: + if not path.is_file() or path.is_symlink(): + raise FileNotFoundError(name) + handle = os.fdopen(os.open(path, os.O_RDONLY | os.O_NOFOLLOW), "rb") + except OSError: + return self._json_response({"ok": False, "error": {"code": "not_found", + "message": f"no capture named {name!r}"}}, 404) + with handle: + snapshot = self._snapshot_capture(handle) + if snapshot is None: + return self._json_response({"ok": False, "error": {"code": "unavailable", + "message": f"capture {name!r} kept being rewritten while it was copied; " + "ask again"}}, 503) + size = os.fstat(snapshot.fileno()).st_size + + def chunks(): + with snapshot: + while chunk := snapshot.read(self.CAPTURE_READ_BYTES): + yield chunk + + return StreamingResponse(chunks(), media_type=PCAP_MEDIA_TYPE, + headers={"Content-Disposition": f'attachment; filename="{name}"', + "Content-Length": str(size)}) + + def _snapshot_capture(self, handle: Any) -> Any | None: + """A copy of the open capture ``handle`` in an unlinked temp file, read back from its start, or None when each + of ``CAPTURE_SNAPSHOT_ATTEMPTS`` copies found the file shorter than what it had copied: tcpdump truncated it to + reuse it, so the copy would mix two rounds of packets. The copy is gone once it's closed.""" + for _ in range(self.CAPTURE_SNAPSHOT_ATTEMPTS): + handle.seek(0) + copy = tempfile.TemporaryFile() + copied, rewritten = 0, False + while chunk := handle.read(self.CAPTURE_READ_BYTES): + copy.write(chunk) + copied += len(chunk) + if os.fstat(handle.fileno()).st_size < copied: + rewritten = True + break + if not rewritten and os.fstat(handle.fileno()).st_size >= copied: + copy.seek(0) + return copy + copy.close() + return None + def _serve_trajectory(self) -> Response: """Stream the whole env trajectory as JSONL; only a mid-write tail line can arrive torn.""" try: @@ -1507,21 +1777,48 @@ def _call_never_reached_child(exc: BaseException) -> bool: return isinstance(exc, McpError) and exc.error.code == 32600 -def reserved_site_path(path: str) -> bool: - """Whether ``path``, as a site's server would route it, is its data plane or its env card.""" - normalized = posixpath.normpath("/" + path.lstrip("/")) - return normalized == RPC_PATH or normalized.startswith(RPC_PATH + "/") or normalized == WELL_KNOWN_PATH +def _normalized(path: str) -> str: + return posixpath.normpath("/" + path.lstrip("/")) -def reserved_site_request(request: Request) -> bool: - """Whether a request on a site's route reaches its data plane or env card: by the path as routed here, or by the - one its server receives, which is the raw path forwarded, its dot segments removed by httpx and then decoded. The - two differ when an encoded slash sits beside a dot segment (``public%2Fx/../agentenv``). A request-target the proxy - can't build a URL for isn't reserved here: ``forward()`` refuses it.""" - if reserved_site_path("/" + request.path_params.get("path", "")): - return True +def reserved_site_path(path: str, reserved: frozenset[str] = frozenset()) -> bool: + """Whether ``path``, as a site's server would route it, is its data plane or its env card, or one of the paths + ``reserved`` names (``site_reserved_paths``), or under one.""" + normalized = _normalized(path) + return normalized == WELL_KNOWN_PATH or any(normalized == base or normalized.startswith(base.rstrip("/") + "/") + for base in (RPC_PATH, *reserved)) + + +def site_reserved_paths(card: dict) -> frozenset[str]: + """The paths a site's card advertises for agentenv, which its route never reaches: its ``url``, and every + extension's endpoint and each of its methods'; each one a path of its own, not the site's root.""" + paths = [card.get("url")] + for ext in (card.get("capabilities") or {}).get("extensions") or []: + params = ext.get("params") if isinstance(ext, dict) else None + if not isinstance(params, dict): + continue + paths += [endpoint for *_, endpoint in operations(params)] + paths.append(params.get("endpoint")) + return frozenset(_normalized(p) for p in paths if isinstance(p, str) and p.startswith("/") and _normalized(p) != "/") + + +def site_request_path(request: HTTPConnection) -> str | None: + """The path a site's server receives for a request on its route: the raw path forwarded, its dot segments removed + by httpx and then decoded. It differs from the path as routed here when an encoded slash sits beside a dot segment + (``public%2Fx/../api`` reaches ``/api``). None for a request-target the proxy can't build a URL for: ``forward()`` + refuses it.""" try: - sent = httpx.URL(upstream_url("http://site", request)).path + return httpx.URL(upstream_url("http://site", request)).path except (ValueError, httpx.InvalidURL): - return False - return reserved_site_path(sent) + return None + + +def reserved_site_request(request: HTTPConnection, reserved: frozenset[str] = frozenset()) -> bool: + """Whether a request or WebSocket on a site's route reaches its data plane or env card, or one of the paths + ``reserved`` names (``site_reserved_paths``): by the path as routed here, or by the one its server receives + (``site_request_path``). A request-target the proxy can't build a URL for isn't reserved here: ``forward()`` + refuses it.""" + if reserved_site_path("/" + request.path_params.get("path", ""), reserved): + return True + sent = site_request_path(request) + return sent is not None and reserved_site_path(sent, reserved) diff --git a/src/agent_env/env/gateway/requirements.txt b/src/agent_env/env/gateway/requirements.txt index 89a76f21d..261328984 100644 --- a/src/agent_env/env/gateway/requirements.txt +++ b/src/agent_env/env/gateway/requirements.txt @@ -3,3 +3,4 @@ httpx mcp>=1.25,<2 pydantic>=2.0 tzdata +websockets>=13 diff --git a/src/agent_env/env/gateway/rest_proxy.py b/src/agent_env/env/gateway/rest_proxy.py index 0812440ec..e863187aa 100644 --- a/src/agent_env/env/gateway/rest_proxy.py +++ b/src/agent_env/env/gateway/rest_proxy.py @@ -10,34 +10,55 @@ round-trip through str), minus the hop-by-hop set, in both directions; - the role header is the gateway's to write: the client's copies are dropped and the role the gateway attributed the request to goes in their place, and a POST to a backing MCP server's own endpoint gets the same treatment - inside its JSON-RPC ``params._meta``; + inside its JSON-RPC ``params._meta``; so is the role key, which goes upstream only beside a role other than + ``default`` the key verified, so a site holding the same key believes the role; +- the upstream learns the address the client reached it at: ``X-Forwarded-Host`` and ``X-Forwarded-Proto`` (a front's + own, when one before the gateway sent them, else the request's ``Host`` and scheme) and ``X-Forwarded-Prefix``, the + route prefix (``/svc/{service}`` or ``/website/{service}``) the gateway strips, which a client never sets: a client + that serves the route at a root of its own (a computer's site proxy, at ``https://notes.test/``) says so with + ``AgentEnv-Site-Root``, and the upstream is then told no prefix; - the upstream response and the client it came through are closed when the response ends, iterated or not. +A WebSocket on ``/website/{service}/{path}`` is relayed under the same rules (:func:`relay_websocket`): the gateway +opens one of its own to the service, with the headers an HTTP request would carry upstream, and relays each message +and the close, either side's ending ending both. + A backing server's env card names its endpoints relative to itself; :func:`rewrite_child_card` roots them under ``/svc/{key}`` so they resolve against the env's address, keeping the gateway's own routes where the gateway offers -the same operation. +the same operation. A website's card names its routes on ``urn:agentenv:website/v1``; :func:`website_routes` reads +them so :func:`route_name` can name the route a proxied request reached. """ from __future__ import annotations +import asyncio +import contextlib import copy import json import logging import posixpath +import re from collections.abc import AsyncIterator, Awaitable, Callable -from typing import Any +from typing import Any, NamedTuple from urllib.parse import unquote import httpx -from starlette.requests import Request +from starlette.requests import HTTPConnection, Request from starlette.responses import Response, StreamingResponse from starlette.types import Receive, Scope, Send +from starlette.websockets import WebSocket, WebSocketState +from websockets.asyncio.client import ClientConnection, connect as connect_websocket +from websockets.exceptions import ConnectionClosed, InvalidHandshake, InvalidStatus # Relative import — gateway runs as a standalone container with no agent_env.* from .constants import ( AGENT_ENV_ROLE_HEADER, + AGENT_ENV_ROLE_KEY_HEADER, AGENT_ENV_ROLE_META_KEY, AGENT_ENV_SESSION_META_KEY, + AGENT_ENV_SITE_ROOT_HEADER, CARD_FETCH_TIMEOUT_S, + DEFAULT_ROLE, + EXT_WEBSITE_URI, WELL_KNOWN_PATH, ) @@ -51,11 +72,27 @@ # The role header as it appears in the raw ASGI header list (lower-cased by the server). ROLE_HEADER_BYTES = AGENT_ENV_ROLE_HEADER.lower().encode() +ROLE_KEY_HEADER_BYTES = AGENT_ENV_ROLE_KEY_HEADER.lower().encode() + +FORWARDED_HOST, FORWARDED_PROTO, FORWARDED_PREFIX = "x-forwarded-host", "x-forwarded-proto", "x-forwarded-prefix" +_FORWARDED = frozenset(name.encode() for name in (FORWARDED_HOST, FORWARDED_PROTO, FORWARDED_PREFIX, + AGENT_ENV_SITE_ROOT_HEADER.lower())) +# A WebSocket's handshake fields, which the gateway's own connection upstream writes for itself. +_WEBSOCKET_HANDSHAKE = frozenset({"sec-websocket-key", "sec-websocket-version", "sec-websocket-extensions", + "sec-websocket-protocol"}) +# What the gateway's own WebSocket handshake answer writes for itself, so a site's copies aren't relayed. +_WEBSOCKET_ANSWER_DROP = HOP_BY_HOP | {b"date", b"server", b"content-length", b"sec-websocket-accept", + b"sec-websocket-protocol", b"sec-websocket-extensions"} +_WEBSOCKET_REFUSAL_DROP = HOP_BY_HOP | {b"date", b"server", b"content-length"} +_HTTP_SCHEME = {"ws": "http", "wss": "https"} +WEBSOCKET_MESSAGE_MAX_BYTES = 16 * 1024 * 1024 async def forward(request: Request, base_url: str, *, role: str, timeout: httpx.Timeout, mcp_server: bool, - on_done: Callable[[int], Awaitable[None]] | None = None) -> Response: - """Forward `request` to the service at `base_url` as `role` and stream the response through. `mcp_server` says + role_key: str | None = None, on_done: Callable[[int], Awaitable[None]] | None = None, + withheld: frozenset[bytes] = frozenset()) -> Response: + """Forward `request` to the service at `base_url` as `role`, vouched for by `role_key` when the deploy has one, + without the headers named in `withheld`, and stream the response through. `mcp_server` says the service is a backing MCP server, whose own endpoint takes the role in its JSON-RPC `_meta` as well as in the header. A 400 names a request-target the proxy cannot route; a 502 names an upstream that never answered. `on_done` is awaited with the status once the response has ended, after its last byte when streamed.""" @@ -64,7 +101,7 @@ async def forward(request: Request, base_url: str, *, role: str, timeout: httpx. except ValueError as e: # an escaped slash inside the route prefix; refuse rather than forward a wrong path return await _done(_json_response({"error": str(e)}, 400), on_done) content = await request.body() # /svc and /website request bodies are small (JSON-RPC, forms) - headers = upstream_headers(request, role) + headers = upstream_headers(request, role, role_key, withheld=withheld) if mcp_server and request.method == "POST" and is_child_mcp_path(request.path_params.get("path", "")): # The gateway is the trust boundary for identity: a child's proxied MCP endpoint stays reachable, but the # role in its JSON-RPC `_meta` is the gateway's to write, like the header above. @@ -87,7 +124,7 @@ async def _done(response: Response, on_done: Callable[[int], Awaitable[None]] | return response -def upstream_url(base_url: str, request: Request) -> str: +def upstream_url(base_url: str, request: HTTPConnection) -> str: # Forward the path as received: uvicorn decodes scope["path"] for routing, so Starlette's {path} param # would send %2F, %3F and %23 to the upstream as separators. latin-1 keeps one char per wire byte. url = base_url @@ -101,18 +138,57 @@ def upstream_url(base_url: str, request: Request) -> str: return url -def upstream_headers(request: Request, role: str) -> httpx.Headers: +def upstream_headers(request: HTTPConnection, role: str, role_key: str | None = None, *, + withheld: frozenset[bytes] = frozenset()) -> httpx.Headers: """One entry per raw header, so repeated fields (cookie, x-forwarded-for) keep their multiplicity. The role - header is the gateway's to set: the client's copies are dropped and the role the gateway attributed the - request to goes in their place, so the upstream never reads a value the gateway did not write.""" + and role key headers are the gateway's to set: the client's copies are dropped and the role the gateway + attributed the request to goes in their place, with `role_key` beside any role but `default`, so the upstream + never reads a value the gateway did not write. So are the forwarded-address headers (`forwarded_address`). The + headers named in `withheld` (lower case), the credentials of the provider in front of the gateway, are dropped, + so no service behind it reads them.""" headers = httpx.Headers([(k, v) for k, v in request.headers.raw - if k.lower() not in HOP_BY_HOP and k.lower() != ROLE_HEADER_BYTES]) + if k.lower() not in HOP_BY_HOP and k.lower() not in (ROLE_HEADER_BYTES, ROLE_KEY_HEADER_BYTES) + and k.lower() not in _FORWARDED and k.lower() not in withheld]) headers[AGENT_ENV_ROLE_HEADER] = role + if role_key and role != DEFAULT_ROLE: + headers[AGENT_ENV_ROLE_KEY_HEADER] = role_key + headers.update(forwarded_address(request)) # Bytes are forwarded undecoded, so only ask upstream for encodings the caller can take. headers.setdefault("accept-encoding", "identity") return headers +def forwarded_address(request: HTTPConnection) -> dict[str, str]: + """Where the client reached the proxied service: the host and scheme it addressed (as a front before the gateway + names them in its own ``X-Forwarded-Host``/``X-Forwarded-Proto``, else the request's ``Host`` and scheme; the + client could set ``Host`` anyway, so believing its front names nothing it couldn't), and the route prefix as sent, + or none for a client that says with ``AgentEnv-Site-Root`` it serves the route at a root of its own: a computer's + site proxy, whose browser would otherwise get the route's prefix twice. Like the host, that only changes the links + the client's own answer names; what the route reaches is the path's alone. A header with nothing to say is left + out.""" + host = _first(request.headers.get(FORWARDED_HOST, "")) or request.headers.get("host", "") + proto = _first(request.headers.get(FORWARDED_PROTO, "")).lower() + at_root = bool(request.headers.get(AGENT_ENV_SITE_ROOT_HEADER, "").strip()) + scheme = _HTTP_SCHEME.get(request.url.scheme, request.url.scheme) + address = {FORWARDED_HOST: host, FORWARDED_PROTO: proto if proto in ("http", "https") else scheme, + FORWARDED_PREFIX: "" if at_root else route_prefix(request)} + return {name: value for name, value in address.items() if value} + + +def _first(value: str) -> str: + return value.split(",")[0].strip() + + +def route_prefix(request: HTTPConnection) -> str: + """The path the client reached the proxied service under, as it sent it: the request-target up to the `{path}` + remainder, without its trailing slash, e.g. ``/website/notes``, or under a root path ``/env/website/notes``.""" + raw_path = request.scope.get("raw_path") or request.scope["path"].encode() + decoded_rest = request.path_params.get("path", "") + if decoded_rest: + raw_path = raw_path[:-len(raw_remainder(raw_path, request.scope["path"], decoded_rest))] + return raw_path.decode("latin-1").rstrip("/") + + def is_child_mcp_path(decoded_rest: str) -> bool: """Whether a /svc/{service}/{path} remainder reaches the backing server's MCP endpoint (`mcp` or anything under it), judged after dot-segment normalisation, since httpx collapses `x/../mcp` to `/mcp` on the way upstream. @@ -162,6 +238,112 @@ def raw_remainder(raw_path: bytes, decoded_path: str, decoded_rest: str) -> byte return rest +async def relay_websocket(websocket: WebSocket, base_url: str, *, role: str, role_key: str | None = None, + open_timeout: float = 10.0, withheld: frozenset[bytes] = frozenset()) -> None: + """Relay `websocket` to the service at `base_url` as `role`: a WebSocket of the gateway's own is opened to the same + path, with the headers ``upstream_headers`` sends, asking for the subprotocols the client offered; `websocket` is + accepted with the one the service chose and the headers its answer carried (a ``Set-Cookie`` among them, each + field kept apart), but those the gateway's own answer writes. Each text and binary message is relayed as it came, and either side's + close, with its code, closes the other; a message from the service larger than ``WEBSOCKET_MESSAGE_MAX_BYTES``, the + size uvicorn takes from a client, closes both with 1009 rather than be held whole. A service that refuses the + handshake has its answer relayed as the refusal, its headers as an accepted one's are; one that never answers is a + 502. The headers named in `withheld` don't go upstream, as with ``upstream_headers``.""" + try: + target_url = upstream_url(base_url, websocket) + except ValueError as e: + await refuse_websocket(websocket, _json_response({"error": str(e)}, 400)) + return + headers = [(key, value) for key, value in upstream_headers(websocket, role, role_key, withheld=withheld).multi_items() + if key.lower() not in _WEBSOCKET_HANDSHAKE] + offered = [protocol.strip() for value in websocket.headers.getlist("sec-websocket-protocol") + for protocol in value.split(",") if protocol.strip()] + try: + upstream = await connect_websocket("ws" + target_url.removeprefix("http"), additional_headers=headers, + subprotocols=offered or None, user_agent_header=None, + open_timeout=open_timeout, max_size=WEBSOCKET_MESSAGE_MAX_BYTES) + except InvalidStatus as e: + refusal = Response(content=bytes(e.response.body or b""), status_code=e.response.status_code) + refusal.raw_headers = [*(item for item in refusal.raw_headers if item[0] == b"content-length"), + *_answer_headers(e.response.headers, drop=_WEBSOCKET_REFUSAL_DROP)] + await refuse_websocket(websocket, refusal) + return + except (InvalidHandshake, OSError, TimeoutError) as e: + logger.error(f"WebSocket proxy to {target_url} failed: {e!r}") + await refuse_websocket(websocket, _json_response({"error": f"Proxy request failed: {e!r}"}, 502)) + return + async with upstream: + await websocket.accept(subprotocol=upstream.subprotocol, + headers=_answer_headers(upstream.response.headers, drop=_WEBSOCKET_ANSWER_DROP)) + await _relay_messages(websocket, upstream) + + +def _answer_headers(headers: Any, *, drop: frozenset[bytes]) -> list[tuple[bytes, bytes]]: + """A site's handshake answer's headers as the gateway relays them, each field apart (a ``Set-Cookie`` per cookie), + but those named in `drop`, which the gateway's own answer writes.""" + return [(key.lower(), value) for key, value in + ((key.encode("ascii", "surrogateescape"), value.encode("ascii", "surrogateescape")) + for key, value in headers.raw_items()) if key.lower() not in drop] + + +async def refuse_websocket(websocket: WebSocket, response: Response) -> None: + """Refuse `websocket`'s handshake with `response`, or, on a server that can't send one, a 403.""" + if "websocket.http.response" in websocket.scope.get("extensions", {}): + await websocket.send_denial_response(response) + else: + await websocket.close(1008) + + +async def _relay_messages(websocket: WebSocket, upstream: ClientConnection) -> None: + """Both directions of an accepted WebSocket until either ends, then both closed: the client's close goes upstream + with its code, and the service's to the client.""" + + async def from_client() -> None: + while True: + message = await websocket.receive() + if message["type"] == "websocket.disconnect": + await upstream.close(_sendable(message.get("code"), abnormal=1001), message.get("reason") or "") + return + if message.get("text") is not None: + await upstream.send(message["text"]) + elif message.get("bytes") is not None: + await upstream.send(message["bytes"]) + + async def from_upstream() -> None: + async for data in upstream: + if isinstance(data, str): + await websocket.send_text(data) + else: + await websocket.send_bytes(data) + + tasks = [asyncio.create_task(from_client()), asyncio.create_task(from_upstream())] + try: + await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) + finally: + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + await upstream.close() + if WebSocketState.DISCONNECTED not in (websocket.client_state, websocket.application_state): + sent = upstream.protocol.close_sent + code = sent.code if sent is not None and sent.code == 1009 else upstream.close_code + with contextlib.suppress(RuntimeError, OSError, ConnectionClosed): + await websocket.close(_sendable(code, abnormal=1011), upstream.close_reason or "") + + +def _sendable(code: int | None, *, abnormal: int) -> int: + """A close code one side reported, as one the other may be sent: none (1005) as a normal close, and a closure + with no close frame (1006, 1015) or a code no endpoint may send as `abnormal`. + + >>> [_sendable(c, abnormal=1011) for c in (None, 1005, 1000, 1006, 4001, 1004)] + [1000, 1000, 1000, 1011, 4001, 1011] + """ + if code in (None, 1005): + return 1000 + if 1000 <= code <= 1003 or 1007 <= code <= 1014 or 3000 <= code <= 4999: + return code + return abnormal + + class UpstreamResponse(StreamingResponse): """Streams an httpx response through untouched and closes it when the response ends, iterated or not. @@ -326,5 +508,69 @@ def operations(params: dict) -> list[tuple[str | None, dict | None, dict, str, s if isinstance(holder.get("endpoint"), str) and holder["endpoint"]] +class WebsiteRoute(NamedTuple): + """One route a website's card names: its verb, the pattern its path matches (a Starlette converter's own, one + path segment for any other converter or none), and its name, None for a route the site builds in.""" + + method: str + pattern: re.Pattern + name: str | None + + +_CONVERTERS = {"path": ".*", "int": "[0-9]+", "float": r"[0-9]+(?:\.[0-9]+)?", + "uuid": "[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}"} +_PATH_PARAM = re.compile(r"{\w+(?::(\w+))?}") + + +def website_routes(card: dict | None) -> list[WebsiteRoute]: + """The routes a website's card names on ``urn:agentenv:website/v1``, in its order: each one's method, its path + (``/api/notes/{note_id}``, with or without a Starlette converter such as ``{note_id:int}``) and its name. An entry + missing any of the three is skipped; a card naming none, or no card, has none. + + Before them come, unnamed, the built-in routes the card names, which answer before any of its routes would: a GET + to its ``health`` path, and a GET or DELETE to its ``session_path`` unless one of its routes is at that very path + (then the site serves its own there). A request one of them answers is named by no route. + + >>> [(r.method, r.pattern.pattern, r.name) for r in website_routes({"capabilities": {"extensions": [ + ... {"uri": "urn:agentenv:website/v1", "params": {"health": "/healthz", "routes": [ + ... {"method": "get", "path": "/api/notes/{note_id:int}", "name": "get_note"}, {"path": "/x"}]}}]}})] + [('GET', '/healthz', None), ('GET', '/api/notes/(?:[0-9]+)', 'get_note')] + """ + found: list[WebsiteRoute] = [] + for ext in ((card or {}).get("capabilities") or {}).get("extensions") or []: + params = ext.get("params") if isinstance(ext, dict) and ext.get("uri") == EXT_WEBSITE_URI else None + routes = params.get("routes") if isinstance(params, dict) else None + named = [route for route in (routes if isinstance(routes, list) else []) + if isinstance(route, dict) and all(isinstance(route.get(k), str) and route[k] + for k in ("method", "path", "name"))] + health, session = (params.get(k) if isinstance(params, dict) else None for k in ("health", "session_path")) + if isinstance(health, str) and health: + found.append(WebsiteRoute("GET", _path_pattern(health), None)) + if isinstance(session, str) and session and all(route["path"] != session for route in named): + found += [WebsiteRoute(verb, _path_pattern(session), None) for verb in ("GET", "DELETE")] + found += [WebsiteRoute(route["method"].upper(), _path_pattern(route["path"]), route["name"]) for route in named] + return found + + +def _path_pattern(path: str) -> re.Pattern: + parts, end = [], 0 + for param in _PATH_PARAM.finditer(path): + parts += [re.escape(path[end:param.start()]), f"(?:{_CONVERTERS.get(param.group(1) or '', '[^/]+')})"] + end = param.end() + return re.compile("".join(parts) + re.escape(path[end:])) + + +def route_name(routes: list[WebsiteRoute], method: str, path: str) -> str | None: + """The name of the first of `routes` a `method` request to `path` reaches (a HEAD reaches a GET route), else None, + as it is when that first one is a built-in route. + + >>> route_name(website_routes({"capabilities": {"extensions": [{"uri": "urn:agentenv:website/v1", "params": { + ... "routes": [{"method": "GET", "path": "/api/notes/{note_id}", "name": "get_note"}]}}]}}), "HEAD", "/api/notes/7") + 'get_note' + """ + verbs = {method.upper(), "GET"} if method.upper() == "HEAD" else {method.upper()} + return next((route.name for route in routes if route.method in verbs and route.pattern.fullmatch(path)), None) + + def _json_response(data: dict | list, status_code: int) -> Response: return Response(content=json.dumps(data), status_code=status_code, media_type="application/json") diff --git a/src/agent_env/env/gateway/triggers.py b/src/agent_env/env/gateway/triggers.py index 1017d3ef3..69dbf0d20 100644 --- a/src/agent_env/env/gateway/triggers.py +++ b/src/agent_env/env/gateway/triggers.py @@ -713,7 +713,8 @@ def _normalize_action_when(when: dict, tid: str) -> None: _normalize_provoked_when(when, tid) def _normalize_website_when(self, when: dict, tid: str) -> None: - """Validate a website `when`: a known app, upper-cased method, `path`/`status` predicates, `where`, `repeat`.""" + """Validate a website `when`: a known app, upper-cased method, the `op` (a route the site's card names), + `path`/`status` predicates, `where`, `repeat`.""" w = f"trigger '{tid}'" app = when.get("app") if not isinstance(app, str) or not app: @@ -726,6 +727,17 @@ def _normalize_website_when(self, when: dict, tid: str) -> None: raise TriggerError(f"{w}: when.method must be a non-empty HTTP method, e.g. 'POST'") else: when["method"] = when["method"].strip().upper() + if when.get("op") is None: + when.pop("op", None) + elif not isinstance(when["op"], str) or not when["op"]: + raise TriggerError(f"{w}: when.op must be a non-empty string naming one of the website's routes") + else: + names = self._gw.website_route_names(app) + if names is None: + raise TriggerError(f"{w}: when.op names a route of website {app!r}, whose card the gateway could not " + "read; match its path instead") + if when["op"] not in names: + raise TriggerError(f"{w}: when.op names no route of website {app!r} (routes: {sorted(names)})") for key in ("path", "status"): if when.get(key) is None: when.pop(key, None) @@ -921,17 +933,18 @@ def on_tool_call(self, role: str, name: str, arguments: dict, return pending, timeout_s def on_website_request(self, role: str, app: str, method: str, path: str, status: int, - args: dict) -> tuple[list[asyncio.Task], float | None]: - """Detection hook for an answered website request, before its response goes out; returns what on_tool_call does.""" + args: dict, op: str | None = None) -> tuple[list[asyncio.Task], float | None]: + """Detection hook for an answered website request, before its response goes out, `op` naming the route of + the site's card it reached; returns what on_tool_call does.""" provokable = [(tid, trig) for tid, trig in self._triggers.items() if self._provoked_by(trig, role)] if not provokable: return [], None if self._has_time_triggers: self._eval_time_triggers() - request = {"app": app, "method": method, "path": path, "status": status} + request = {"app": app, "method": method, "path": path, "status": status, **({"op": op} if op else {})} provoking = {"source": "website", **request, "role": role} matched = [(tid, trig) for tid, trig in provokable - if _website_matches(trig["spec"]["when"], app, method, path, status)] + if _website_matches(trig["spec"]["when"], app, method, path, status, op)] pending, timeout_s = self._detect(matched, provoking, {"args": args, "request": request}) if method not in _SAFE_METHODS and status < 400: self._reevaluate_states(provokable, provoking) @@ -963,10 +976,12 @@ def _reevaluate_states(self, provokable: list[tuple[str, dict]], provoking: dict trig["evaluating"] = True self._schedule(self._eval_state(tid, trig, provoking)) - def may_hold_website_request(self, role: str, app: str, method: str, path: str, status: int) -> bool: + def may_hold_website_request(self, role: str, app: str, method: str, path: str, status: int, + op: str | None = None) -> bool: """Whether a `provoking_call` barrier could hold this website request: a matcher check, `where` and status aside.""" return any((trig["spec"].get("barrier") or {}).get("at") == "provoking_call" and self._provoked_by(trig, role) - and _website_matches(trig["spec"]["when"], app, method, path, status) for trig in self._triggers.values()) + and _website_matches(trig["spec"]["when"], app, method, path, status, op) + for trig in self._triggers.values()) @property def has_website_triggers(self) -> bool: @@ -1608,10 +1623,13 @@ def _normalize_provoked_when(when: dict, tid: str) -> None: raise TriggerError(f"trigger '{tid}': when.repeat must be a boolean") -def _website_matches(when: dict, app: str, method: str, path: str, status: int) -> bool: - """Whether a request matches a website `when` (`where` aside); without `status`, only a response below 400 does.""" +def _website_matches(when: dict, app: str, method: str, path: str, status: int, op: str | None = None) -> bool: + """Whether a request matches a website `when` (`where` aside); with `op`, only one reaching that route does, and + without `status`, only a response below 400 does.""" if when["type"] != "website" or when["app"] != app or when.get("method", method) != method: return False + if "op" in when and when["op"] != op: + return False if "path" in when and not _eval_predicate(when["path"], path): return False return _eval_predicate(when["status"], status) if "status" in when else status < 400 diff --git a/src/agent_env/env/legacy_protocol.py b/src/agent_env/env/legacy_protocol.py index 16899025b..df302b5e9 100644 --- a/src/agent_env/env/legacy_protocol.py +++ b/src/agent_env/env/legacy_protocol.py @@ -8,11 +8,13 @@ from __future__ import annotations import asyncio +import dataclasses import logging -from typing import TYPE_CHECKING, Optional +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, Optional import httpx -from agentenv_protocol import RPC_PATH, DataPart, client as protocol_v1 +from agentenv_protocol import METHOD_GET, DataPart, client as protocol_v1 if TYPE_CHECKING: from agent_env.env.env import DeployedEnv @@ -33,27 +35,141 @@ def environment_base_url(gateway_url: Optional[str], environment_name: str, mcp: return f"{gateway_url}/svc/{prefix}{environment_name}" +@dataclass(frozen=True) +class BackingServer: + """A server behind an env's address, by its environment_name: an MCP server or, with ``mcp=False``, a website. + + One the env's card lists (``card_servers``) carries that ``child`` and the ``address`` its paths resolve against. + One the card lists none for, a server serving no card, carries the ``address`` alone: it has no card, and none is + read for it. One named by the env definition of a record whose card can't be read carries neither, so its card is + read live at its gateway path, ``/svc/mcp-{name}`` or a website's ``/svc/{name}`` (``server_card``).""" + + environment_name: str + mcp: bool = True + child: Optional[protocol_v1.ChildEnv] = None + address: Optional[str] = None + + @property + def card(self) -> Optional[dict]: + """The server's card as the env's card lists it, or None.""" + return self.child.card if self.child is not None else None + + +def definition_servers(env: Any) -> list[BackingServer]: + """The servers an env definition names: a MultiEnv's MCP servers then its websites, a WebsiteEnv's website, or + else the env's one MCP server; empty for an env with none. Read only for what the env's card can't say + (``backing_servers``).""" + from agent_env.env.envs.website import WebsiteEnv + + if isinstance(env, WebsiteEnv): + return [BackingServer(env.environment_name, mcp=False)] + servers = [BackingServer(e.environment_name) for e in getattr(env, "mcp_server_envs", None) or []] + servers += [BackingServer(e.environment_name, mcp=False) for e in getattr(env, "website_envs", None) or []] + if not servers and getattr(env, "environment_name", None): + servers = [BackingServer(env.environment_name)] + return servers + + +def card_servers(card: dict, address: str) -> list[BackingServer]: + """The servers ``card``, read at ``address``, lists, each with its own card: a leaf card's env itself; else each + child the composed card nests, after the env itself when its own card serves ``data/get``, as a computer's driver + fronting its apps does. A child is a website when its card carries ``website/v1`` or its url is at a website's + gateway route, ``/svc/{name}`` (an MCP server's is ``/svc/mcp-{name}``), so a website and an MCP server sharing a + name are two servers.""" + own = protocol_v1.ChildEnv(name=card.get("name"), url=card.get("url"), card=card) + if not isinstance(card.get("children_environments"), list): + return [_listed(own, address)] + servers = [_listed(own, address)] if protocol_v1.serves(card, METHOD_GET) else [] + return servers + [_listed(child, address) for child in protocol_v1.children(card)] + + +def _listed(child: protocol_v1.ChildEnv, address: str) -> BackingServer: + from agent_env.env.env import website_child + + return BackingServer(child.name, mcp=not website_child(child), child=child, address=address) + + +async def backing_servers(deployed: Optional[DeployedEnv], gateway_url: Optional[str], env: Any, + timeout: int = 10) -> list[BackingServer]: + """The servers behind an env as its card lists them (``card_servers``), then, behind a gateway, each server its + definition ``env`` names that the card lists none of the same name and kind for: one serving no card, which the + gateway leaves off its card, so it has none. + + The card is ``deployed``'s stored one when ``gateway_url`` is its own gateway or None, else read live at + ``gateway_url`` (an override, or a record that kept no card). A record with no card URL, an older one, or no + record, or a live read that fails, leaves the definition alone: its servers, each read live at its gateway path, + logged once.""" + from agent_env.env.env import gateway_url_of + + gateway = gateway_url.rstrip("/") if gateway_url else None + card, address, why = None, None, "it has no record" + if deployed is not None and deployed.environment_card and (gateway is None or (gateway_url_of(deployed) or "").rstrip("/") == gateway): + card, address = deployed.environment_card, deployed.environment_url + elif deployed is not None and deployed.environment_card_url and gateway: + try: + card, address = await protocol_v1.get_card(gateway, timeout=timeout), gateway + except httpx.HTTPError as e: + why = f"its card at {gateway} could not be read ({type(e).__name__})" + elif deployed is not None: + why = "its record has no card URL" + named = definition_servers(env) + if not isinstance(card, dict): + logger.info(f"env {getattr(env, 'id', None)!r}: {why}, so its servers are the ones its definition names, " + f"each read live at its gateway path: {[(s.environment_name, s.mcp) for s in named]}") + return named + servers = card_servers(card, address) + if not gateway: + return servers + listed = {(s.environment_name, s.mcp) for s in servers} + return servers + [dataclasses.replace(s, address=address) for s in named if (s.environment_name, s.mcp) not in listed] + + +async def server_card(server: BackingServer, gateway_url: Optional[str], timeout: int = 10) -> tuple[Optional[str], Optional[dict]]: + """A server's card and the base its endpoints resolve against: as the env's card lists it, None for a server the + card lists none for, else read live at its gateway path (``child_env_card``).""" + if server.address is not None: + return server.address, server.card + return await child_env_card(None, gateway_url, server.environment_name, timeout=timeout, mcp=server.mcp) + + +async def server_v1_base_url(server: BackingServer, gateway_url: Optional[str]) -> Optional[str]: + """A server's v1 data-plane base: its card's ``url`` resolved against the env's address, None for a server the + env's card lists none for, else as the live probe at its gateway path finds it (``v1_base_url``).""" + if server.address is not None: + return server.child.base_url(server.address) if server.child is not None else None + return await v1_base_url(None, gateway_url, server.environment_name, mcp=server.mcp) + + async def v1_base_url(deployed: Optional[DeployedEnv], gateway_url: Optional[str], environment_name: str, mcp: bool = True) -> Optional[str]: - """A child env's v1 data-plane base from the stored env card, or None for legacy; without a stored card, the live probe decides.""" + """A child env's v1 data-plane base from the stored env card, the MCP server's or with ``mcp=False`` the website's + of that name, or None for legacy; without a stored card, the live probe + decides. A website's probe is a card read that is None only on a 404 and raises on any other failure, so a website whose + card could not be read is never taken for one serving no data plane.""" if deployed is None or not deployed.environment_card: base_url = environment_base_url(gateway_url, environment_name, mcp=mcp) + if not mcp: + _, card = await child_env_card(None, gateway_url, environment_name, mcp=False) + return base_url if card is not None else None return base_url if await protocol_v1.supports_v1(base_url) else None - child = deployed.get_child_env_card(environment_name) - return deployed.environment_url + child.get("url", RPC_PATH).removesuffix(RPC_PATH) if child else None + child = deployed.child_env(environment_name, mcp=mcp) + return child.base_url(deployed.environment_url) if child is not None else None -async def child_env_card(deployed: Optional[DeployedEnv], gateway_url: Optional[str], environment_name: str, timeout: int = 10) -> tuple[str, Optional[dict]]: - """An MCP child env's card and the base its endpoints resolve against: from the stored env card, or without one a live read (None on 404).""" +async def child_env_card(deployed: Optional[DeployedEnv], gateway_url: Optional[str], environment_name: str, timeout: int = 10, + mcp: bool = True) -> tuple[str, Optional[dict]]: + """A child env's card and the base its endpoints resolve against: from the stored env card, the MCP server's or + with ``mcp=False`` the website's of that name, or without one a live read + at the MCP server's gateway path, or with ``mcp=False`` the website's (None on 404).""" if deployed is None or not deployed.environment_card: - base_url = environment_base_url((gateway_url or "").rstrip("/"), environment_name) + base_url = environment_base_url((gateway_url or "").rstrip("/"), environment_name, mcp=mcp) try: return base_url, await protocol_v1.get_card(base_url, timeout=timeout) except httpx.HTTPStatusError as e: if e.response.status_code == 404: return base_url, None raise - return deployed.environment_url, deployed.get_child_env_card(environment_name) + return deployed.environment_url, deployed.get_child_env_card(environment_name, mcp=mcp) async def reset_via_rest( base_url: str, diff --git a/src/agent_env/providers/env_providers/env_computer_provider.py b/src/agent_env/providers/env_providers/env_computer_provider.py index 1d0b996c5..40b7be6d6 100644 --- a/src/agent_env/providers/env_providers/env_computer_provider.py +++ b/src/agent_env/providers/env_providers/env_computer_provider.py @@ -5,16 +5,19 @@ import functools import logging -from datetime import datetime, timezone +from datetime import datetime, timedelta, timezone from typing import TYPE_CHECKING, ClassVar, Optional from agentenv_protocol.computer import COMPUTER_DISPLAY, COMPUTER_RECORDING +from agentenv_protocol.computer.app import COMPUTER_APP +from agentenv_protocol.computer.app_runtime import HOLD_BOOT_LAUNCH_ENV from agent_env.artifact.artifacts.computer import ComputerArtifact from agent_env.attribution import Attribution from agent_env.env.env import DeployedSandboxEnv, EnvCapabilityUnsupported from agent_env.env.envs.computer.utils import params as computer_params from agent_env.env.envs.computer.utils.probing import probe_first_boot +from agent_env.env.envs.computer.utils import sites from agent_env.env.envs.computer.utils.boot import ( app_settings_environment, boot, @@ -36,6 +39,7 @@ if TYPE_CHECKING: from agent_env.env.env import Env from agent_env.env.envs.computer.env import ComputerEnv + from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import SandboxProvider logger = logging.getLogger(__name__) @@ -74,10 +78,16 @@ async def deploy( params, values, problems = computer_params.checked(env, env_params) if problems: raise DeployRefused(f"{env.id}: " + "; ".join(problems)) + if params.site_routing and not params.sites: + raise DeployRefused(f"{env.id}: env_params.site_routing reaches the run's websites, which a run's " + "deploy_env names in env_params.sites; give them there to deploy it any other way") await env.compose_unstarted(sandbox_provider, params.form) if not (builds := env.deployable_builds(params.form)): raise DeployRefused(f"{env.id} has no ready build to deploy; its builds are " f"{env.build_statuses() or 'none yet'}; compose it with: agent-env env build --id {env.id}") + if params.site_routing and not (builds := [build for build in builds if build[0] != "disk"]): + raise DeployRefused(f"{env.id}: env_params.site_routing installs into the computer's container, and a disk " + "computer runs inside a launcher container it can't reach; deploy a container form") attempt = functools.partial(self._boot, env, builds, params, values, ttl_seconds=ttl_seconds, disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, attribution=attribution) return await self._run(sandbox_provider, env.id, attempt) @@ -132,6 +142,9 @@ async def _boot(self, env: ComputerEnv, builds: list[tuple[str, str, dict]], par "env_params": params.model_dump(exclude_defaults=True)}}, ) await _require_display(record, values.get("resolution")) + if params.site_routing: + record.metadata["computer"]["sites"] = await _route_sites(env, booted.sandbox, params, ttl_seconds) + await _launch_held_apps(env, record, booted.url) except BaseException: await discard(booted.sandbox) raise @@ -161,6 +174,29 @@ async def reboot(self, env: ComputerEnv, deployed: DeployedSandboxEnv) -> None: deployed.control_token) _require_name(env, card) await _require_display(deployed, values.get("resolution")) + if params.site_routing: + record["sites"] = await _route_sites(env, sandbox, params, record["sites"]["valid_for_s"]) + await _launch_held_apps(env, deployed, deployed.environment_url) + + +async def _route_sites(env: ComputerEnv, sandbox: Sandbox, params: computer_params.ComputerParams, + valid_for_s: int) -> dict: + """The computer on ``sandbox`` routed to ``params.sites`` under a CA of its own, valid ``valid_for_s``; what its + record keeps of it.""" + routed = await sites.route(sandbox, params.sites, user=env.computer.session.user, network=env.network, + valid_for=timedelta(seconds=valid_for_s), label=env.environment_name) + return {**routed, "valid_for_s": valid_for_s} + + +async def _launch_held_apps(env: ComputerEnv, record: DeployedSandboxEnv, url: str) -> None: + """Launch each app that starts with the computer, whose launch a deploy routing sites holds until the sites' names, + CA and proxy are in place, through its ``computer-app/v1`` ``launch``, then wait until the computer is ready.""" + os_type = env.computer.os.type + held = [app for app in env.composed_apps() if (launch := app.launch(os_type)) is not None and launch.at == "boot"] + for app in held: + await record.invoke(COMPUTER_APP, "launch", environment_name=app.environment_name) + if held: + await ready(url, ready_timeout(env.composed_apps(), os_type), record.control_token) async def end_recordings(record: DeployedSandboxEnv, reason: str) -> None: @@ -185,7 +221,8 @@ def _environment(env: ComputerEnv, computer: ComputerArtifact, values: dict, return {**settings_environment(computer, values, environment_name=env.environment_name, driver_settings=params.driver_settings), **app_settings_environment(env.composed_apps(), computer_params.app_settings(env, params.app_settings), - env.computer.os.type)} + env.computer.os.type), + **({HOLD_BOOT_LAUNCH_ENV: "1"} if params.site_routing else {})} def _require_name(env: ComputerEnv, card: dict) -> None: diff --git a/src/agent_env/providers/env_providers/env_gateway_provider.py b/src/agent_env/providers/env_providers/env_gateway_provider.py index 84ca8cb11..ee7aa6f28 100644 --- a/src/agent_env/providers/env_providers/env_gateway_provider.py +++ b/src/agent_env/providers/env_providers/env_gateway_provider.py @@ -7,6 +7,7 @@ import json import logging import re +import secrets import shlex import time from collections import Counter @@ -15,17 +16,20 @@ from typing import TYPE_CHECKING, Optional import httpx -from agentenv_protocol import client as protocol_v1 +from agentenv_protocol.website import WEBSITE_EXTENSION_URI, WEBSITE_READY_OP, website_readiness from agent_env.artifact.artifacts.docker_image import pull_problem from agent_env.attribution import Attribution from agent_env.env.env import DeployedGatewayEnv, _mcp_url from agent_env.env.gateway import AGENT_ENV_GATEWAY_MCP_PORT, GatewayMode from agent_env.env.gateway.constants import ( + CONTROL_TOKEN_ENV, DEFAULT_MCP_SERVER_NAME, - EXT_WEBSITE_URI, + GATEWAY_PROVIDER_HEADERS_ENV, + GATEWAY_ROLE_KEY_ENV, STATE_TOOL_NAME_CLASHES_KEY, - WEBSITE_READY_OP, + TRAFFIC_CAPTURE_DIR_ENV, + WEBSITE_ROLE_KEY_ENV, WELL_KNOWN_PATH, random_mcp_server_name, tool_name_clashes_text, @@ -62,6 +66,7 @@ SANDBOX_MODE_VM, Accepts, SandboxProvider, + sandbox_request_header_names, ) from agent_env.store.image_store.oci_registry_credentials import image_reference_problem @@ -80,6 +85,7 @@ # How much of a service's output a failed deploy's error carries: enough for the end of a traceback. _LOG_TAIL_LINES = 30 _LOG_LINE_CHARS = 300 +_CAPTURE_CHECK_ERROR_CHARS = 500 # Gathering it may stall with the connection; past this, the deploy raises the error it had. _DOWN_SERVICES_TIMEOUT_S = 60.0 @@ -92,6 +98,13 @@ WEBSITE_READY_TIMEOUT_S = 300.0 WEBSITE_READY_POLL_S = 2.0 +TRAFFIC_CAPTURE_SUFFIX = "traffic" +TRAFFIC_VOLUME = "traffic-captures" +TRAFFIC_GATEWAY_DIR = "/var/lib/agentenv/traffic" +TRAFFIC_SIDECAR_DIR = "/captures" +TRAFFIC_CAPTURE_FILE_MB = 20 +TRAFFIC_CAPTURE_FILES = 10 + @dataclass class MCPServerConfig: @@ -119,18 +132,30 @@ class DeployedGateway: # The env card the readiness probe read, and when (ISO 8601, UTC). environment_card: dict | None = None environment_card_read_at_utc: str | None = None + role_key: str | None = None + control_token: str | None = None @dataclass class WebsiteConfig: """Configuration for a website in a gateway deployment: its server image, which serves the site's pages and API and says when it's ready by its card's ``website/v1`` ``ready`` (``health_path``, deprecated, for a site whose card - has none), and a deprecated separate frontend image for a site built on one.""" + has none), a deprecated separate frontend image for a site built on one, and the env's ``settings`` as stored, + secrets as their references, which only the compose file holds resolved.""" backend_image: str frontend_image: str | None environment_name: str health_path: str = AGENT_ENV_WEBSITE_HEALTH_PATH + settings: dict[str, str] | None = None + + def env_vars(self) -> dict[str, str]: + """The settings with each ``secret:`` / ``env:`` reference resolved from the deploying user's config.""" + if not self.settings: + return {} + from agent_env.config import get_config + resolve = get_config().resolve_reference + return {name: resolve(value) for name, value in self.settings.items()} @dataclass @@ -283,6 +308,9 @@ def create_docker_compose( host_ips: tuple[str, ...] = (), extra_hosts: tuple[str, ...] = (), mcp_server_name: str | None = None, + role_key: str | None = None, + capture_traffic: bool = False, + control_token: str | None = None, ) -> str: """Generate docker-compose.yml content for a gateway deployment onto a VM/laptop/arbitrary machine. @@ -294,6 +322,13 @@ def create_docker_compose( gateway_image: Docker image for the gateway. gateway_port: Port to expose for the gateway. website_configs: Optional list of website configurations (frontend + backend pairs). + role_key: The deploy's role key, given to the gateway as AGENTENV_ROLE_KEY and to each website server as + WEBSITE_ROLE_KEY; without one, both believe the AgentEnv-Role header as before it. + capture_traffic: Add a capture service beside each website's server container, and its deprecated + frontend container when it has one (see ``traffic_capture_services``): tcpdump, from the gateway's + image, in that container's network namespace, the only service holding NET_RAW and NET_ADMIN, writing + rotating pcap files to a volume the gateway mounts read-only and serves as traffic/v1. + control_token: The token the gateway's framework-only routes take, given to it as AGENTENV_CONTROL_TOKEN. Returns: docker-compose.yml content as a string. @@ -381,7 +416,8 @@ def create_docker_compose( env-network: driver: bridge """ - problems = _compose_name_problems(mcp_servers, website_configs, sidecars, images=(gateway_image,)) + problems = _compose_name_problems(mcp_servers, website_configs, sidecars, images=(gateway_image,), + capture_traffic=capture_traffic) if mcp_server_name and ("\n" in mcp_server_name or "\r" in mcp_server_name): problems.append(f"MCP server name {mcp_server_name!r} must be one line") if problems: @@ -457,6 +493,9 @@ def create_docker_compose( dep_lines += [f" {name}:", " condition: service_healthy"] for backend_name in website_backend_servers.values(): dep_lines += [f" {backend_name}:", " condition: service_healthy"] + captures = traffic_capture_services(website_configs) if capture_traffic else {} + for capture_name in captures: + dep_lines += [f" {capture_name}:", " condition: service_started"] if dep_lines: lines.append(" depends_on:") lines.extend(dep_lines) @@ -491,6 +530,16 @@ def create_docker_compose( gateway_env.append(f" - REST_PROXY_URLS={','.join(rest_proxy_urls)}") gateway_env.append(f" - GATEWAY_MODE={gateway_mode.value}") gateway_env.append(f" - SERVICE_DB_URL={state_provider.base_url(state_instance)}") + if role_key: + gateway_env.append(f" - {GATEWAY_ROLE_KEY_ENV}={role_key}") + if provider_headers := sandbox_request_header_names(): + gateway_env.append(f" - {GATEWAY_PROVIDER_HEADERS_ENV}={','.join(provider_headers)}") + if control_token: + gateway_env.append(f" - {CONTROL_TOKEN_ENV}={control_token}") + gateway_volumes = [] + if capture_traffic: + gateway_env.append(f" - {TRAFFIC_CAPTURE_DIR_ENV}={TRAFFIC_GATEWAY_DIR}") + gateway_volumes = [" volumes:", f" - {_traffic_mount(TRAFFIC_GATEWAY_DIR, read_only=True)}"] publish = host_port or (lambda port: port) exposed_gateway_ports = [] if expose_gateway_port: @@ -505,6 +554,7 @@ def create_docker_compose( *gateway_env, *exposed_gateway_ports, *gateway_extra_hosts, + *gateway_volumes, " restart: on-failure", " healthcheck:", f' test: ["CMD-SHELL", "python3 -c \\"import socket; s=socket.create_connection((\'localhost\',{gateway_port}),2); s.close()\\""]', @@ -536,6 +586,8 @@ def create_docker_compose( f" - ENVIRONMENT_NAME={wc.environment_name}", f" - PORT={AGENT_ENV_WEBSITE_BACKEND_PORT}", " - PYTHONUNBUFFERED=1", + *([f" - {WEBSITE_ROLE_KEY_ENV}={role_key}"] if role_key else []), + *(f" - {compose_env_entry(name, value)}" for name, value in wc.env_vars().items()), " healthcheck:", f' test: ["CMD-SHELL", "curl -sf -o /dev/null http://localhost:{AGENT_ENV_WEBSITE_BACKEND_PORT}{WELL_KNOWN_PATH}' f' || curl -sf -o /dev/null http://localhost:{AGENT_ENV_WEBSITE_BACKEND_PORT}{wc.health_path}"]', @@ -560,6 +612,9 @@ def create_docker_compose( "", ]) + for capture_name, container in captures.items(): + lines.extend([f" {capture_name}:", *_traffic_capture_service(gateway_image, container), ""]) + # Add sidecar services. # Secrets go through the compose environment block, never a docker-run command line. for sc in sidecars or []: @@ -586,6 +641,8 @@ def create_docker_compose( " env-network:", " driver: bridge", ]) + if capture_traffic: + lines.extend(["volumes:", f" {TRAFFIC_VOLUME}: {{}}"]) return "\n".join(lines) async def deploy( @@ -603,17 +660,25 @@ async def deploy( attribution: Optional[Attribution] = None, artifact_id: str | None = None, artifact_version: int | None = None, + keyed_roles: bool = False, + capture_traffic: bool = False, ) -> DeployedGatewayEnv: """A gateway in front of the env, with the state store it acquires (local Postgres by default); returns the record. A MultiEnv's gateway fronts its MCP servers and websites, a WebsiteEnv's its website, and any other env's its one MCP server. + With ``keyed_roles``, the deploy draws a role key (see ``create_gateway``). With ``capture_traffic``, it captures + its websites' traffic (see ``create_gateway``), and refuses an env that serves no website. """ from agent_env.config import get_config, service_db_env_id from agent_env.env.env import Env from agent_env.providers.env_state import LOCAL_POSTGRES_STATE_TYPE, acquire_state_for_deploy topology = _gateway_topology(env) - if problems := _compose_name_problems(topology.mcp_servers, topology.website_configs): + if capture_traffic and not topology.website_configs: + raise ValueError(f"Can't capture {env.id!r}'s traffic: capture_traffic records what a website's containers " + "send and receive, and the env serves no website") + if problems := _compose_name_problems(topology.mcp_servers, topology.website_configs, + capture_traffic=capture_traffic): raise ValueError(f"Can't deploy {env.id!r} behind a gateway: {'; '.join(problems)}") state_instance = await acquire_state_for_deploy( env_state_type=env_state_type, ttl_seconds=ttl_seconds, name_hint=env.id, env_state_instance_id=env_state_instance_id, @@ -625,6 +690,7 @@ async def deploy( mcp_server_images=topology.mcp_server_images, website_configs=topology.website_configs, website_images=topology.website_images, mcp_server_name=topology.mcp_server_name, gateway_mode=gateway_mode, ttl_seconds=ttl_seconds, disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, attribution=attribution, + keyed_roles=keyed_roles, **({"capture_traffic": True} if capture_traffic else {}), ) return DeployedGatewayEnv( env_id=env.id, @@ -641,6 +707,8 @@ async def deploy( website_frontend_urls=deployed.website_frontend_urls, gateway_mode=gateway_mode.value, env_state_instance_ids=deployed.env_state_instance_ids, + role_key=deployed.role_key, + control_token=deployed.control_token, ) async def create_gateway( @@ -663,6 +731,8 @@ async def create_gateway( mcp_server_name: str | None = None, *, attribution: Optional[Attribution] = None, + keyed_roles: bool = False, + capture_traffic: bool = False, ) -> DeployedGateway: """Create and start a gateway from explicit topology, for a caller that builds its own record rather than calling ``deploy()``. @@ -676,20 +746,33 @@ async def create_gateway( ``mcp_server_name`` is the name the gateway presents on its card and as its MCP server; drawn as ``env`` + 4 random digits when the env declares none; the record takes it from the gateway's card. + + ``keyed_roles`` draws a secret role key for the deploy, given to the gateway and each website server and kept on + the record: the gateway then believes a role other than ``default`` only beside it, so every caller acting as one + must send it. Off, as before the key, the role header alone is believed. + + ``capture_traffic`` records every packet each website's server and frontend containers send and receive once + its capture attaches, just after the container starts, with tcpdump in a capture service beside each, as rotating pcap files the gateway serves as traffic/v1, behind a + control token the deploy draws for it and keeps on the record (``control_token``). VM deploys only, since only + they serve websites; off, the stack is as before it. """ return await self._launch( sandbox_provider, env_id, state_instance=state_instance, mcp_servers=mcp_servers, mcp_server_images=mcp_server_images, gateway_port=gateway_port, website_configs=website_configs, website_images=website_images, gateway_mode=gateway_mode, ttl_seconds=ttl_seconds, disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, existing_sandbox=existing_sandbox, sidecars=sidecars, mcp_server_name=mcp_server_name, - attribution=attribution, + attribution=attribution, keyed_roles=keyed_roles, **({"capture_traffic": True} if capture_traffic else {}), ) - async def _launch(self, sandbox_provider: SandboxProvider, env_id: str | None, *, mcp_server_name: str | None = None, **gateway) -> DeployedGateway: - """One gateway, deployed on the sandbox provider or each chained member in turn, then its tools probed.""" + async def _launch(self, sandbox_provider: SandboxProvider, env_id: str | None, *, mcp_server_name: str | None = None, + keyed_roles: bool = False, capture_traffic: bool = False, **gateway) -> DeployedGateway: + """One gateway, deployed on the sandbox provider or each chained member in turn, then its tools probed. Its role + key and control token are drawn once, so every chained attempt holds the same.""" attempt = functools.partial( self._deploy_gateway, **gateway, mcp_server_name=mcp_server_name or random_mcp_server_name(), # drawn once, so every chained attempt presents the same name + role_key=secrets.token_urlsafe(32) if keyed_roles else None, + **({"capture_traffic": True, "control_token": secrets.token_urlsafe(32)} if capture_traffic else {}), ) earlier_sandbox = self._sandbox result = await self._run(sandbox_provider, env_id, attempt) @@ -741,6 +824,9 @@ async def _deploy_via_vm( existing_sandbox: Sandbox | None = None, sidecars: list[SidecarConfig] | None = None, mcp_server_name: str | None = None, + role_key: str | None = None, + capture_traffic: bool = False, + control_token: str | None = None, ) -> DeployedGateway: from agent_env.env.env import Env from agent_env.config import gateway_env_id, get_config, website_browser_env_id @@ -832,6 +918,8 @@ async def _stand_up(_spec): host_ips=sandbox.host_ips, extra_hosts=sandbox.extra_hosts, mcp_server_name=mcp_server_name, + role_key=role_key, + **({"capture_traffic": True, "control_token": control_token} if capture_traffic else {}), ) logger.info(f"Generated docker-compose.yml:\n{_redact_compose_secrets(compose_content)}") @@ -880,6 +968,8 @@ async def _stand_up(_spec): if not gateway_ready: down = await self._down_services(sandbox) raise RuntimeError("Gateway did not become ready in time" + (f":\n{down}" if down else "")) + if capture_traffic: + await self._refuse_stopped_captures(sandbox, list(traffic_capture_services(website_configs))) # Check container status only on the sidecars the provider actually rendered sidecar_names = ( @@ -904,7 +994,9 @@ async def _stand_up(_spec): card = await self._wait_for_tunnel(gateway_url, timeout=120) if card is None: raise RuntimeError("Tunnel did not become ready in time") - await _wait_for_websites(gateway_url, website_configs or []) + if website_configs: + await _wait_for_websites(gateway_url, website_configs) + card = await self._wait_for_tunnel(gateway_url, timeout=120) or card card_read_at_utc = datetime.now(timezone.utc).isoformat() await self._refuse_tool_name_clashes(gateway_url) mcp_url = _mcp_url(gateway_url, card) @@ -925,7 +1017,8 @@ async def _stand_up(_spec): gateway_url=gateway_url, mcp_url=mcp_url, db_web_url=db_web_url, db_mcp_url=db_mcp_url, website_frontend_urls=website_frontend_urls, mcp_server_name=mcp_server_name, env_state_instance_ids=[state_instance.instance_id], - environment_card=card, environment_card_read_at_utc=card_read_at_utc, + environment_card=card, environment_card_read_at_utc=card_read_at_utc, role_key=role_key, + control_token=control_token, ) async def _deploy_via_containers( @@ -942,6 +1035,7 @@ async def _deploy_via_containers( memory_mb: int | None = None, attribution: Optional[Attribution] = None, mcp_server_name: str | None = None, + role_key: str | None = None, ) -> DeployedGateway: from agent_env.env.env import Env from agent_env.config import gateway_env_id, get_config @@ -1038,6 +1132,8 @@ async def _deploy_via_containers( "GATEWAY_MODE": gateway_mode.value, "SERVICE_DB_URL": self._state_provider.base_url(state_instance), "MCP_SERVER_NAME": mcp_server_name or DEFAULT_MCP_SERVER_NAME, + **({GATEWAY_ROLE_KEY_ENV: role_key} if role_key else {}), + **({GATEWAY_PROVIDER_HEADERS_ENV: ",".join(names)} if (names := sandbox_request_header_names()) else {}), } gateway_sb = await sandbox_provider.create_container( image_name=gateway_env.docker_image_artifact.image_name, @@ -1071,6 +1167,7 @@ async def _deploy_via_containers( env_state_instance_ids=[state_instance.instance_id], environment_card=card, environment_card_read_at_utc=card_read_at_utc, + role_key=role_key, ) except Exception: await self.close() @@ -1169,6 +1266,32 @@ async def _down_services(self, sandbox: VmSandbox) -> str | None: logger.warning(f"Can't list the stack's services: no answer in {_DOWN_SERVICES_TIMEOUT_S:.0f}s") return None + async def _refuse_stopped_captures(self, sandbox: VmSandbox, capture_services: list[str]) -> None: + """Refuse a deploy whose capture services aren't all running once the gateway is up, as one is whose gateway + image has no tcpdump: it would record nothing. A lookup that fails is a warning, since it says nothing of them.""" + try: + exit_code, stdout, stderr = await sandbox.exec_with_output( + "sudo", "docker", "compose", "-f", DOCKER_COMPOSE_PATH, "ps", "-a", "--format", "json") + except Exception as e: + logger.warning(f"Can't check the traffic captures are running: {e}") + return + if exit_code != 0: + logger.warning(f"Can't check the traffic captures are running: docker compose ps exited {exit_code}: " + f"{stderr.strip()[-_CAPTURE_CHECK_ERROR_CHARS:]}") + return + rows = {row.get("Service"): row for row in _compose_ps_rows(stdout)} + stopped = [] + for name in capture_services: + row = rows.get(name) + if row is None or row.get("State") != "running": + tail = await self._log_tail(sandbox, str((row or {}).get("ID") or "")) + why = _why_down(row) if row is not None else "has no container" + stopped.append(f"{name} {why or row.get('State')}" + (f":\n{tail}" if tail else "")) + if stopped: + raise RuntimeError("The env's traffic captures aren't running, so its traffic wouldn't be recorded (a gateway " + "image built before captures has no tcpdump: put the gateway env again):\n" + + "\n".join(stopped)) + async def _read_down_services(self, sandbox: VmSandbox) -> str | None: try: exit_code, stdout, stderr = await sandbox.exec_with_output( @@ -1372,9 +1495,12 @@ async def _deploy_gateway( sidecars: list[SidecarConfig] | None = None, mcp_server_name: str | None = None, attribution: Optional[Attribution] = None, + role_key: str | None = None, + capture_traffic: bool = False, + control_token: str | None = None, ) -> DeployedGateway: """One gateway deploy on one (unchained) sandbox provider: a VM on one that creates VMs, else a container per - server on its private network.""" + server on its private network, which captures no traffic.""" from agent_env.providers.env_state import LocalPostgresStateProvider, build_state_provider # Every attempt starts from the caller's store: a failed attempt's close() cleared it. @@ -1386,6 +1512,9 @@ async def _deploy_gateway( ) deploy_kwargs = {"attribution": dict(attribution or {})} if not self.in_vm(sandbox_provider): + if capture_traffic: + raise ValueError(f"Can't capture traffic on {type(sandbox_provider).__name__}, which runs the gateway in " + "containers: a capture runs beside its website's container on the gateway's VM") if problem := self.sandbox_problem(sandbox_provider, websites=bool(website_configs)): raise ValueError(f"Can't deploy a gateway on {type(sandbox_provider).__name__}, which {problem}") if existing_sandbox is not None: @@ -1396,7 +1525,7 @@ async def _deploy_gateway( sandbox_provider, mcp_servers, mcp_server_images, gateway_port=gateway_port, website_configs=website_configs, gateway_mode=gateway_mode, ttl_seconds=ttl_seconds, disk_size_gb=disk_size_gb, - cpu=cpu, memory_mb=memory_mb, mcp_server_name=mcp_server_name, **deploy_kwargs, + cpu=cpu, memory_mb=memory_mb, mcp_server_name=mcp_server_name, role_key=role_key, **deploy_kwargs, ) earlier_sandbox = self._sandbox try: @@ -1405,7 +1534,8 @@ async def _deploy_gateway( gateway_port=gateway_port, website_configs=website_configs, website_images=website_images, gateway_mode=gateway_mode, ttl_seconds=ttl_seconds, disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, existing_sandbox=existing_sandbox, sidecars=sidecars, - mcp_server_name=mcp_server_name, **deploy_kwargs, + mcp_server_name=mcp_server_name, role_key=role_key, **deploy_kwargs, + **({"capture_traffic": True, "control_token": control_token} if capture_traffic else {}), ) except BaseException: await self._terminate_new_sandbox(earlier_sandbox=earlier_sandbox, existing_sandbox=existing_sandbox) @@ -1501,12 +1631,72 @@ class _GatewayTopology: names_gateway_on_vm: bool # a MultiEnv's record names its gateway sandbox even on a VM +def compose_env_entry(name: str, value: str) -> str: + """``name=value`` as one quoted YAML scalar of a compose ``environment:`` list, so the value stays that variable's + whatever it holds: a newline, quote or ``: `` can't start more of the service, and each ``$`` is ``$$``, which + compose doesn't interpolate. The scalar is plain ASCII, any other character written as its escape.""" + escaped = [] + for char in f"{name}={value}".replace("$", "$$"): + if char in '"\\': + escaped.append("\\" + char) + elif " " <= char <= "~": + escaped.append(char) + elif ord(char) <= 0xFFFF: + escaped.append(f"\\u{ord(char):04x}") + else: + escaped.append(f"\\U{ord(char):08x}") + return '"' + "".join(escaped) + '"' + + +def traffic_capture_services(website_configs: list[WebsiteConfig] | None) -> dict[str, str]: + """The capture service a deploy capturing its websites' traffic runs for each website container, by name, each + with the compose service of the container whose network namespace it captures in: every website's server, and its + deprecated frontend when it has one.""" + containers = [f"{wc.environment_name}-{suffix}" for wc in website_configs or [] + for suffix in (AGENT_ENV_WEBSITE_BACKEND_SUFFIX, AGENT_ENV_WEBSITE_FRONTEND_SUFFIX) + if suffix == AGENT_ENV_WEBSITE_BACKEND_SUFFIX or wc.frontend_image] + return {f"{container}-{TRAFFIC_CAPTURE_SUFFIX}": container for container in containers} + + +def _traffic_mount(path: str, *, read_only: bool = False) -> str: + """The compose ``volumes:`` entry mounting ``TRAFFIC_VOLUME`` at ``path``.""" + return f"{TRAFFIC_VOLUME}:{path}" + (":ro" if read_only else "") + + +def _traffic_capture_service(gateway_image: str, container: str) -> list[str]: + """The compose lines of the service capturing ``container``'s traffic: tcpdump, from the gateway's image, on every + interface of the container's, its loopback among them, every packet whole and written as it's captured, rotating + through ``TRAFFIC_CAPTURE_FILES`` files of ``TRAFFIC_CAPTURE_FILE_MB`` million bytes named for the container, its + oldest overwritten once they're full, so a long run holds at most their product per container on the VM's disk. It joins + the container's network namespace, so it sees what the container sends and receives and nothing else, from when it + attaches, which is just after the container starts: what a server sends while it starts up can come before it, and + isn't recorded. It is the only service given the capabilities a capture needs. It writes to ``TRAFFIC_VOLUME``, which the gateway mounts + read-only and no website container mounts at all.""" + argv = ["-i", "any", "-n", "-s", "0", "-U", "-Z", "root", "-C", str(TRAFFIC_CAPTURE_FILE_MB), + "-W", str(TRAFFIC_CAPTURE_FILES), "-w", f"{TRAFFIC_SIDECAR_DIR}/{container}.pcap"] + return [ + f" image: {gateway_image}", + f' network_mode: "service:{container}"', + " depends_on:", + f" - {container}", + " cap_add:", + " - NET_RAW", + " - NET_ADMIN", + ' entrypoint: ["tcpdump"]', + f" command: {json.dumps(argv)}", + " volumes:", + f" - {_traffic_mount(TRAFFIC_SIDECAR_DIR)}", + " restart: unless-stopped", + ] + + def website_config(env: WebsiteEnv) -> WebsiteConfig: """A website env as the gateway runs it: its server image, any deprecated frontend image, and its health path.""" frontend = env.frontend_docker_image_artifact return WebsiteConfig(backend_image=env.server_docker_image_artifact.image_name, frontend_image=frontend.image_name if frontend else None, - environment_name=env.environment_name, health_path=env.health_path) + environment_name=env.environment_name, health_path=env.health_path, + settings=dict(env.settings) or None) def _gateway_topology(env: Env) -> _GatewayTopology: @@ -1547,10 +1737,12 @@ def environment_name_problem(name: str) -> str | None: def _compose_name_problems(mcp_servers: list[MCPServerConfig], website_configs: list[WebsiteConfig] | None, - sidecars: list[SidecarConfig] | None = None, *, images: tuple[str, ...] = ()) -> list[str]: + sidecars: list[SidecarConfig] | None = None, *, images: tuple[str, ...] = (), + capture_traffic: bool = False) -> list[str]: """Why the environment names and images of a gateway deploy can't go into its compose file and shell lines: one can't be a service name, two of its services would share one (a website's are suffixed, so a website may share an - MCP server's name), or one is a service every gateway deploy runs.""" + MCP server's name; so are the capture services of a deploy with ``capture_traffic``), or one is a service every + gateway deploy runs.""" names = [c.environment_name for c in [*mcp_servers, *(website_configs or []), *(sidecars or [])]] refs = [*images, *(s.image for s in mcp_servers), *(s.image for s in sidecars or []), *(ref for w in website_configs or [] for ref in (w.backend_image, w.frontend_image) if ref is not None)] @@ -1558,7 +1750,8 @@ def _compose_name_problems(mcp_servers: list[MCPServerConfig], website_configs: if problem] services = Counter([*(c.environment_name for c in [*mcp_servers, *(sidecars or [])]), *(f"{w.environment_name}-{suffix}" for w in website_configs or [] - for suffix in (AGENT_ENV_WEBSITE_BACKEND_SUFFIX, AGENT_ENV_WEBSITE_FRONTEND_SUFFIX))]) + for suffix in (AGENT_ENV_WEBSITE_BACKEND_SUFFIX, AGENT_ENV_WEBSITE_FRONTEND_SUFFIX)), + *(traffic_capture_services(website_configs) if capture_traffic else ())]) problems += [f"two of its services would both be named {name!r}" for name, count in services.items() if count > 1] problems += [f"environment name {name!r} is one a gateway deploy names its own containers " f"({', '.join(sorted(GATEWAY_SERVICE_NAMES))})" for name in services if name in GATEWAY_SERVICE_NAMES] @@ -1568,7 +1761,8 @@ def _compose_name_problems(mcp_servers: list[MCPServerConfig], website_configs: # Compose `environment:` keys whose values are secrets and must be masked before the # rendered compose is logged. Matches the key to the left of the first '=' on an # ` - KEY=VALUE` env line. -_SECRET_ENV_KEY_RE = re.compile(r"^(\s*-\s*)([A-Za-z0-9_]*(?:SECRET|PASSWORD|TOKEN|API_KEY)[A-Za-z0-9_]*)=.*$") +_SECRET_ENV_KEY_RE = re.compile(r"^(\s*-\s*)([A-Za-z0-9_]*(?:SECRET|PASSWORD|TOKEN|API_KEY|ROLE_KEY)[A-Za-z0-9_]*)=.*$") +_SETTING_ENV_RE = re.compile(r'^(\s*-\s*")([A-Za-z_][A-Za-z0-9_]*)=.*"$') def _compose_ps_rows(text: str) -> list[dict]: @@ -1599,9 +1793,12 @@ def _why_down(row: dict) -> str | None: def _redact_compose_secrets(compose_content: str) -> str: - """Mask secret-valued env lines so the logged compose never carries a live secret.""" + """Mask secret-valued env lines so the logged compose never carries a live secret, and every website setting's + quoted ``"NAME=VALUE"`` line as ``compose_env_entry`` writes it, whatever its name: its value may be a credential + a ``secret:`` or ``env:`` reference resolved.""" return "\n".join( - _SECRET_ENV_KEY_RE.sub(r"\1\2=***", line) for line in compose_content.splitlines() + _SETTING_ENV_RE.sub(r'\1\2=***"', _SECRET_ENV_KEY_RE.sub(r"\1\2=***", line)) + for line in compose_content.splitlines() ) @@ -1632,7 +1829,7 @@ async def _wait_for_websites(gateway_url: str, website_configs: list[WebsiteConf ready, why[name] = await _website_ready(f"{gateway_url}/svc/{name}", wc.health_path) if ready is None and name not in warned: warned.add(name) - logger.warning(f"website {name!r} has no {EXT_WEBSITE_URI} {WEBSITE_READY_OP} on its card, so its " + logger.warning(f"website {name!r} has no {WEBSITE_EXTENSION_URI} {WEBSITE_READY_OP} on its card, so its " f"readiness is read at its health path {wc.health_path}, which is deprecated") if ready is not False: del waiting[name] @@ -1645,18 +1842,17 @@ async def _wait_for_websites(gateway_url: str, website_configs: list[WebsiteConf async def _website_ready(base_url: str, health_path: str) -> tuple[Optional[bool], str]: - """Whether the site at ``base_url`` is ready and, when not, why: True or False by its card's ``website/v1`` - ``ready``; with no card or no ``ready`` on it, None once its health path answers 200 and False before.""" + """Whether the site at ``base_url`` is ready and, when not, why: True or False by the ``WebsiteReadiness`` its + card's ``website/v1`` ``ready`` answers, as the protocol's ``website_readiness`` reads it, an answer that isn't one + False; with no card or no ``ready`` on it, None once its health path answers 200 and False before.""" async with httpx.AsyncClient() as client: try: card = _card_of(await client.get(base_url + WELL_KNOWN_PATH, timeout=10), base_url) - if card is not None and protocol_v1.find_extension_method(card, EXT_WEBSITE_URI, WEBSITE_READY_OP): - answer = await protocol_v1.invoke_extension(base_url, card, EXT_WEBSITE_URI, method=WEBSITE_READY_OP, - timeout=10) - if isinstance(answer, dict) and answer.get("ready") is True: + readiness = None if card is None else await website_readiness(base_url, card, timeout=10) + if readiness is not None: + if readiness.ready: return True, "" - problems = answer.get("problems") if isinstance(answer, dict) else None - return False, ", ".join(map(str, problems)) if problems else f"{WEBSITE_READY_OP} says not ready" + return False, ", ".join(readiness.problems) or f"{WEBSITE_READY_OP} says not ready" health = await client.get(base_url + health_path, timeout=10) except (httpx.HTTPError, ValueError) as e: return False, f"{type(e).__name__}: {e}" diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index 4246ad97d..ddd9cc80a 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -621,6 +621,15 @@ def sandbox_request_headers_for_url(url: str) -> dict[str, str]: return headers +def sandbox_request_header_names() -> tuple[str, ...]: + """The names of every header any registered provider attaches to requests for its URLs + (``sandbox_request_headers_for_url``), lower-cased and sorted: what a gateway takes off the requests it forwards, + so no service behind it reads a provider's credentials. Names only, so no credential is resolved.""" + names = {header.lower() for cls in registered_sandbox_provider_classes() + for entries in (cls.REQUEST_HEADERS or {}).values() for header in entries} + return tuple(sorted(names)) + + def _resolve_container_env(provider_cls: type[SandboxProvider]) -> dict[str, str]: """``provider_cls.CONTAINER_ENV`` resolved to values. diff --git a/src/agent_env/task/episode/rules.py b/src/agent_env/task/episode/rules.py index 10f3d78c8..875d09946 100644 --- a/src/agent_env/task/episode/rules.py +++ b/src/agent_env/task/episode/rules.py @@ -148,12 +148,13 @@ def _rubrics_verifier(step: RubricsVerifierTaskStep, outside: Outside) -> list[s UNAFFECTED = frozenset({ # They name no agent. - "aggregate_verifiers", "apply_server_config", "build_mcp_cli", "combine_universe_verdicts", "configure_scheduled_checks", + "aggregate_verifiers", "apply_server_config", "build_mcp_cli", "collect_traffic_capture", "combine_universe_verdicts", + "configure_scheduled_checks", "deploy_env", "deploy_sandbox", "end_computer_recording", "start_computer_recording", "env_outcome_verifier", "modify_env_tool_access", "reset_env", "review", - "run_container_unit_tests_verifier", "run_docker_container", "snapshot_env", "snapshot_computer_env", "sync_env_clock", + "run_container_unit_tests_verifier", "run_docker_container", "sign_in_browser", "snapshot_env", "snapshot_computer_env", "sync_env_clock", "validation_gate_aggregator", "verify_env_card", "verify_env_core_protocol", "verify_mcp_tool_schema", - "verify_spec_conformance", "verify_universe_load_export_roundtrip", "website_state_verifier", + "verify_spec_conformance", "verify_universe_load_export_roundtrip", # They read only what an outside turn leaves: its reply. "agent_prompt_response_verifier", "verify_mcp_env_assessment", # It skips an agent that isn't deployed. diff --git a/src/agent_env/task/episode/run.py b/src/agent_env/task/episode/run.py index 2b082cd69..5ee2a6214 100644 --- a/src/agent_env/task/episode/run.py +++ b/src/agent_env/task/episode/run.py @@ -283,9 +283,11 @@ def _raise_unless_completed(self) -> None: if (error := self._run.exception()) is not None: raise error - async def _serve_in_process(self, agent: str, role: str | None, mcp_urls: dict[str, str]) -> DeployedAgent: + async def _serve_in_process(self, agent: str, role: str | None, mcp_urls: dict[str, str], + role_keys: dict[str, str | None] | None = None) -> DeployedAgent: """A proxy served in this process for ``agent``, which works in no sandbox, with each env server registered as - ``deploy_agent`` registers them with any agent, at the URL this process reaches: played as the others are.""" + ``deploy_agent`` registers them with any agent, at the URL this process reaches, with its deploy's role key + from ``role_keys``: played as the others are.""" server = InProcessProxy() self._in_process.append(server) url = await server.start() @@ -295,7 +297,8 @@ async def _serve_in_process(self, agent: str, role: str | None, mcp_urls: dict[s card = response.json() mcp_config = A2AAgent.find_extension(card, A2AAgent.EXT_MCP_CONFIG) for env_id, mcp_url in mcp_urls.items(): - body = _mcp_add_body(mcp_config, mcp_url, sandbox_request_headers_for_url(mcp_url), env_id, role) + body = _mcp_add_body(mcp_config, mcp_url, sandbox_request_headers_for_url(mcp_url), env_id, role, + (role_keys or {}).get(env_id)) (await client.post(url + mcp_config["params"]["endpoint"], json=body)).raise_for_status() deployed = DeployedAgent(agent_name=agent, api_url=url, a2a_url=url, a2a_card=card, role=role) await self._attach(agent, deployed) diff --git a/src/agent_env/task/episode/steps.py b/src/agent_env/task/episode/steps.py index 9bcc85816..7b8c12e51 100644 --- a/src/agent_env/task/episode/steps.py +++ b/src/agent_env/task/episode/steps.py @@ -45,11 +45,12 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: episode = _episode_of(context) agent = self.step.agent_name if not self.sandboxed: - mcp_urls = {} + mcp_urls, role_keys = {}, {} for env_id in getattr(self.step, "env_ids", None) or []: deployed = bound_deployment(context, env_id, env_step_id=self.step.env_step_id, step_id=self.id) mcp_urls[env_id] = deployed.mcp_url if deployed is not None else live_mcp_url(env_id) - deployed = await episode._serve_in_process(agent, getattr(self.step, "role", None), mcp_urls) + role_keys[env_id] = deployed.role_key if deployed is not None else None + deployed = await episode._serve_in_process(agent, getattr(self.step, "role", None), mcp_urls, role_keys) context.deployed_agents.append(deployed) if system_prompt := getattr(self.step, "system_prompt", None): config = A2AAgent.negotiate_agent_config(deployed.a2a_card, {"system_prompt": system_prompt}) diff --git a/src/agent_env/task/store.py b/src/agent_env/task/store.py index 869db45d0..1a720a82a 100644 --- a/src/agent_env/task/store.py +++ b/src/agent_env/task/store.py @@ -789,6 +789,12 @@ def record_task_cancelled_sync( ), ) + def set_context_metadata_sync(self, instance_id: str, key: str, value: object) -> None: + """Set ``context.metadata[key]`` of a stored instance outside any step, as teardown records what it did after + the run's last step saved; not journaled, since nothing after teardown undoes it.""" + self._doc_store.update(TASK_INSTANCES_COLLECTION, Filter.of(instance_id=instance_id), + UpdateSpec(set={f"context.metadata.{key}": value})) + def append_step_attempt_failure_sync(self, instance_id: str, failure: "StepAttemptFailure") -> None: """Best-effort append of one attempt's failure, designed not to conflict with the ``record_task_failure`` write. ``add_to_set`` creates the array on the first append and @@ -898,6 +904,15 @@ def seed_task_instance_context(instance_id: str, context: "TaskStepContext") -> logger.warning("Failed to seed task instance context", exc_info=True) +def record_context_metadata(instance_id: str, key: str, value: object) -> None: + """Save ``context.metadata[key]`` of the stored instance ``instance_id``, as teardown does once the run's steps + have saved theirs; a failure is logged, never raised.""" + try: + get_task_instance_store().set_context_metadata_sync(instance_id, key, value) + except Exception: + logger.warning("Failed to save context.metadata[%r] of task instance %s", key, instance_id, exc_info=True) + + async def record_step_complete( instance_id: str, step_id: str, diff --git a/src/agent_env/task/task.py b/src/agent_env/task/task.py index 6840a58d8..c6ce57022 100644 --- a/src/agent_env/task/task.py +++ b/src/agent_env/task/task.py @@ -18,6 +18,7 @@ from typing import TYPE_CHECKING, Callable, ClassVar, Optional from agentenv_protocol import client as protocol_v1 +from agentenv_protocol.traffic import serves_traffic from agent_env.config import run_env_ids from agent_env.plugins import _registration @@ -44,7 +45,9 @@ def _rolled_back_sandboxes(before: dict, after: dict) -> dict: context ``before`` an undo and absent ``after`` it, shaped like the context's own slices so the executor's cleanup can scan the marker exactly as it scans a context. The undo is what removes these from the live and persisted context, so without this record an - orphaned sandbox from the failed attempt is discoverable nowhere. + orphaned sandbox from the failed attempt is discoverable nowhere. ``capturing_envs`` keeps + the whole record of each env among them that serves traffic/v1, so teardown can still + collect its captures before its sandboxes go. """ def gone(key: str) -> list[dict]: kept = {_identity(e) for e in after.get(key) or []} @@ -64,6 +67,7 @@ def gone(key: str) -> list[dict]: {"sandbox_id": x.get("sandbox_id"), "sandbox_type": x.get("sandbox_type")} for x in gone("deployed_sandboxes") ], + "capturing_envs": [e for e in gone("deployed_envs") if serves_traffic(e.get("environment_card") or {})], } diff --git a/src/agent_env/task/teardown.py b/src/agent_env/task/teardown.py index 9f11627c4..2468b70c1 100644 --- a/src/agent_env/task/teardown.py +++ b/src/agent_env/task/teardown.py @@ -3,7 +3,8 @@ ``teardown_run`` takes down everything a run's context records: its bare sandboxes, its envs and its agents. It removes compute only; the instance, its outputs and the artifacts it collected stay. What has to reach the stores goes first: the changelogs staged on its agents (``before_teardown``), the recordings the run left open, ended with fresh -grants whatever env serves them, and what each env saves as it closes (``Env.close``); a recording still open after +grants whatever env serves them, the traffic captures of each env deployed with ``capture_traffic`` that the run didn't +collect, and what each env saves as it closes (``Env.close``); a recording still open after that is registered from what its namespace holds. Then every sandbox is terminated by id, and a local one's work folder goes once it's down. ``agent-env run`` and ``agent-env eval run`` call it after each run, and the ``teardown_sandboxes`` step takes down part of a run the same way (``tear_down``). ``Task.run`` never does, so a @@ -31,6 +32,7 @@ from agent_env.task.interrupts import tearing_down from agent_env.task_step.context import TaskStepContext from agent_env.task_step.task_steps.computer_recording import end_open_recordings, register_open_recordings +from agent_env.task_step.task_steps.traffic_capture import collect_uncollected_captures, rolled_back_deployments logger = logging.getLogger(__name__) @@ -116,7 +118,9 @@ async def teardown_run(context: TaskStepContext, *, timeout: float = TERMINATE_T async def tear_down(context: TaskStepContext, sandboxes: Sequence[RecordedSandbox], envs: Sequence[DeployedEnv], *, timeout: float) -> TeardownReport: - """End the recordings the run left open on ``envs`` with fresh grants, close each env still up, register any + """End the recordings the run left open on ``envs`` with fresh grants, collect the traffic captures of each of them + that captures and that the run didn't collect, and of each capturing env a retry rolled back whose sandboxes are + among ``sandboxes``, close each env still up, register any recording still open from its namespace, then terminate each of ``sandboxes`` by id, all at once and each within ``timeout`` seconds, and remove a local one's work folder once it's down. A close that fails is logged: its env's sandboxes are terminated by id regardless, which leaves one the close took down as it is. Cancelling it returns what it reached, with the rest in ``left``.""" @@ -129,8 +133,15 @@ async def tear_down(context: TaskStepContext, sandboxes: Sequence[RecordedSandbo await end_open_recordings(context, env_ids) except TimeoutError: logger.warning("teardown: the run's open recordings weren't all ended within %gs", timeout) - await asyncio.gather(*(_close(env, timeout) for env in envs - if any(sandbox_id in up for sandbox_id, _ in env_sandbox_ids(env)))) + still_up = [env for env in envs if any(sandbox_id in up for sandbox_id, _ in env_sandbox_ids(env))] + rolled_back = [(key, env) for key, env in rolled_back_deployments(context) + if any(sandbox_id in up for sandbox_id, _ in env_sandbox_ids(env))] + try: + async with asyncio.timeout(timeout): + await collect_uncollected_captures(context, still_up, rolled_back=rolled_back) + except TimeoutError: + logger.warning("teardown: the run's uncollected traffic captures weren't all collected within %gs", timeout) + await asyncio.gather(*(_close(env, timeout) for env in still_up)) await asyncio.to_thread(register_open_recordings, context, env_ids) except asyncio.CancelledError: return TeardownReport(left=pending) diff --git a/src/agent_env/task_step/registry.py b/src/agent_env/task_step/registry.py index a4457a320..f482a64be 100644 --- a/src/agent_env/task_step/registry.py +++ b/src/agent_env/task_step/registry.py @@ -39,6 +39,7 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]: from agent_env.task_step.task_steps.deploy_sandbox import DeploySandboxTaskStep from agent_env.task_step.task_steps.reset_env import ResetEnvTaskStep from agent_env.task_step.task_steps.teardown_sandboxes import TeardownSandboxesTaskStep + from agent_env.task_step.task_steps.traffic_capture import CollectTrafficCaptureTaskStep from agent_env.task_step.task_steps.verifiers.env_outcome_verifier import EnvOutcomeVerifierTaskStep from agent_env.task_step.task_steps.load_artifact import LoadArtifactTaskStep from agent_env.task_step.task_steps.modify_env_tool_access import ModifyEnvToolAccessStep @@ -47,10 +48,10 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]: from agent_env.task_step.task_steps.register_agent_triggers import RegisterAgentTriggersStep from agent_env.task_step.task_steps.sync_env_clock import SyncEnvClockTaskStep from agent_env.task_step.task_steps.configure_scheduled_checks import ConfigureScheduledChecksTaskStep + from agent_env.task_step.task_steps.sign_in_browser import SignInBrowserTaskStep from agent_env.task_step.task_steps.install_agent import InstallAgentTaskStep from agent_env.task_step.task_steps.verifiers.rubrics_verifier import RubricsVerifierTaskStep from agent_env.task_step.task_steps.verifiers.run_container_unit_tests_verifier import RunContainerUnitTestsVerifierTaskStep - from agent_env.task_step.task_steps.verifiers.website_state_verifier import WebsiteStateVerifierTaskStep from agent_env.task_step.task_steps.run_docker_container import RunDockerContainerTaskStep from agent_env.task_step.task_steps.mcp_env_validator.verify_mcp_tool_schema import VerifyMCPToolSchemaTaskStep from agent_env.task_step.task_steps.mcp_env_validator.verify_spec_conformance import VerifySpecConformanceTaskStep @@ -91,6 +92,7 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]: _get_type(CollectArtifactsTaskStep): CollectArtifactsTaskStep, _get_type(StartComputerRecordingTaskStep): StartComputerRecordingTaskStep, _get_type(EndComputerRecordingTaskStep): EndComputerRecordingTaskStep, + _get_type(CollectTrafficCaptureTaskStep): CollectTrafficCaptureTaskStep, _get_type(DeployAgentTaskStep): DeployAgentTaskStep, _get_type(DeployEnvTaskStep): DeployEnvTaskStep, _get_type(DeployHumanAgentTaskStep): DeployHumanAgentTaskStep, @@ -105,10 +107,10 @@ def _builtin_registry() -> dict[str, type["TaskStep"]]: _get_type(RegisterAgentTriggersStep): RegisterAgentTriggersStep, _get_type(SyncEnvClockTaskStep): SyncEnvClockTaskStep, _get_type(ConfigureScheduledChecksTaskStep): ConfigureScheduledChecksTaskStep, + _get_type(SignInBrowserTaskStep): SignInBrowserTaskStep, _get_type(InstallAgentTaskStep): InstallAgentTaskStep, _get_type(RubricsVerifierTaskStep): RubricsVerifierTaskStep, _get_type(RunContainerUnitTestsVerifierTaskStep): RunContainerUnitTestsVerifierTaskStep, - _get_type(WebsiteStateVerifierTaskStep): WebsiteStateVerifierTaskStep, _get_type(RunDockerContainerTaskStep): RunDockerContainerTaskStep, _get_type(VerifyMCPToolSchemaTaskStep): VerifyMCPToolSchemaTaskStep, _get_type(VerifySpecConformanceTaskStep): VerifySpecConformanceTaskStep, diff --git a/src/agent_env/task_step/task_steps/add_skills.py b/src/agent_env/task_step/task_steps/add_skills.py index 1c9505431..f8fccc3d5 100644 --- a/src/agent_env/task_step/task_steps/add_skills.py +++ b/src/agent_env/task_step/task_steps/add_skills.py @@ -12,7 +12,7 @@ from typing import ClassVar, Optional from agentenv_protocol import SKILL_EXTENSION_URI -from agentenv_protocol.client import find_extension_method +from agentenv_protocol.client import children, find_extension_method from agent_env.config import get_config from agent_env.task_step.context import DeployedAgent, TaskStepContext @@ -327,7 +327,8 @@ def _stored_skill(skill_md: str, files: dict[str, str]) -> str: async def _env_skills(context: TaskStepContext, agent: DeployedAgent, env_id: str) -> list[dict]: """The skills the env ``env_id`` serves, each under its card's environment name: its own card's and each of its - child envs' (a gateway's servers, a computer's apps) its name reaches; refused for an env this run didn't deploy or + child envs' (a gateway's servers, a computer's apps), each asked as its own card advertises it, a name another + already serves skills under skipped; refused for an env this run didn't deploy or bind to ``agent``, or that serves none.""" deployed = next((d for d in context.deployed_envs if d.env_id == env_id), None) if deployed is None: @@ -338,12 +339,13 @@ async def _env_skills(context: TaskStepContext, agent: DeployedAgent, env_id: st card = deployed.environment_card or {} serving = [(None, _env_skill_name(card.get("name") or env_id, env_id))] if find_extension_method( card, SKILL_EXTENSION_URI, "get") is not None else [] - serving += [(child["name"], _env_skill_name(child["name"], f"{env_id}/{child['name']}")) - for child in card.get("children_environments") or [] - if find_extension_method(child, SKILL_EXTENSION_URI, "get") is not None - and deployed.get_child_env_card(child["name"]) is child] + named = {card.get("name")} if serving else set() + for child in children(card): + if find_extension_method(child.card, SKILL_EXTENSION_URI, "get") is not None and child.name not in named: + named.add(child.name) + serving.append((child, _env_skill_name(child.name, f"{env_id}/{child.name}"))) if not serving: raise RuntimeError(f"env '{env_id}' serves no skill: neither its card nor a child env's declares " f"{SKILL_EXTENSION_URI}") - return [await deployed.invoke(SKILL_EXTENSION_URI, "get", {"name": name}, environment_name=child) + return [await deployed.invoke(SKILL_EXTENSION_URI, "get", {"name": name}, child=child) for child, name in serving] diff --git a/src/agent_env/task_step/task_steps/apply_server_config.py b/src/agent_env/task_step/task_steps/apply_server_config.py index 41f82ffd0..91cc24167 100644 --- a/src/agent_env/task_step/task_steps/apply_server_config.py +++ b/src/agent_env/task_step/task_steps/apply_server_config.py @@ -1,22 +1,24 @@ """Task step that applies a task's behavior config to deployed servers. At setup (after the env is deployed + universe loaded, before the agent runs) -this step invokes each backing MCP server's ``set_*`` config extensions over the -gateway. Extensions are advertised on the server's ``EnvironmentCard`` (served at +this step invokes each backing MCP server's or website's ``set_*`` config extensions +over the gateway. Extensions are advertised on the server's ``EnvironmentCard`` (served at ``/.well-known/agent-env.json``) under ``capabilities.extensions[]`` and invoked at their advertised REST endpoint (e.g. ``/agentenv/ext/set_errors``). Config lives on the backing server, not on the gateway itself. Each server's card comes from the env card stored on the deployed record, whose extension endpoints the -gateway has already rewritten to its ``/svc/mcp-{service}`` proxy; a record without a -stored card reads it live there. The agent never sees the card or these endpoints; -only this harness step invokes them. +gateway has already rewritten to its ``/svc/mcp-{service}`` (or a website's ``/svc/{service}``) +proxy; a record without a stored card reads it live there. A broadcast reaches each server +the env card lists, by that server's own card and route, so a website sharing its name with +an MCP server is reached too (``legacy_protocol.backing_servers``). The agent never sees the +card or these endpoints; only this harness step invokes them. """ from __future__ import annotations import logging -from typing import Any, ClassVar, Optional +from typing import TYPE_CHECKING, Any, ClassVar, Optional import httpx from agentenv_protocol import client as protocol_v1 @@ -25,6 +27,9 @@ from agent_env.entity.refs import EntityRef from agent_env.task_step.task_step import TaskStep, TaskStepDependency +if TYPE_CHECKING: + from agent_env.env.legacy_protocol import BackingServer + logger = logging.getLogger(__name__) @@ -76,7 +81,7 @@ class ApplyServerConfigStep(TaskStep): entity_refs = (EntityRef.env("env_id"),) # A directive whose ``service`` is this sentinel is a broadcast: at execution - # it expands to one directive per MCP server in the env (same uri + args). + # it expands to one directive per MCP server and website in the env (same uri + args). # Lets a caller apply an env-wide config (e.g. set_acting_user across every # server) without enumerating services itself. Pairs naturally with # ``tolerate_unadvertised`` so servers that don't opt in are simply skipped. @@ -131,43 +136,39 @@ def from_dict(cls, data: dict) -> "ApplyServerConfigStep": tolerate_unadvertised=data.get("tolerate_unadvertised", False), ) - def _expand_directives(self, deployed) -> list[ConfigDirective]: - """Expand any broadcast directive (service == ``*``) into one directive - per MCP server in the env, resolved from the env definition. The env's - child service names aren't otherwise enumerable here (the deployed card - list carries no roster), so we read them from ``Env.get`` — matching how - deploy_env / load_artifact resolve the env. Non-broadcast directives - pass through untouched.""" + async def _expand_directives(self, deployed) -> list[tuple[ConfigDirective, Optional[BackingServer], bool]]: + """Each directive with the server it targets and whether it is lenient: a broadcast (service ``*``) expanded + into one directive per MCP server and website the env card lists (``legacy_protocol.backing_servers``), each + targeting that server by its own card and route. A directive a broadcast made for a website is lenient: it is + skipped, as under ``tolerate_unadvertised``, when the website has no card or doesn't advertise the URI, so a + website the task didn't name never fails it. A named directive, and every MCP server, stays strict; a named one + targets the server of that name, the MCP server when a website shares it, as a MultiEnv's load does, or, without + a broadcast to list them, is looked up by name (None).""" if not any(d.service == self._BROADCAST for d in self.directives): - return self.directives + return [(d, None, False) for d in self.directives] - from agent_env.env.env import Env + from agent_env.env import legacy_protocol + from agent_env.env.env import Env, gateway_url_of - env = Env.get(deployed.env_id, deployed.env_version) - environment_names = [ - e.environment_name for e in (getattr(env, "mcp_server_envs", None) or []) - ] - # A single-server env (a bare MCPServerEnv) exposes `environment_name` and no - # `mcp_server_envs`; mirror snapshot_env / exporter enumeration and fall - # back to it, else a broadcast silently matches nothing on single-server envs. - if not environment_names and getattr(env, "environment_name", None): - environment_names = [env.environment_name] + servers = await legacy_protocol.backing_servers(deployed, gateway_url_of(deployed), Env.get(deployed.env_id, deployed.env_version), + timeout=self.timeout_seconds) # A broadcast that resolves to zero services would apply nothing while the # step reports success: the exact silent misconfig this step prevents. Fail # loud (tolerate_unadvertised governs advertisement, not missing servers). - if not environment_names: + if not servers: raise ApplyServerConfigError( - f"apply_server_config: broadcast directive matched no MCP servers " - f"in env {self.env_id!r} (env exposes neither mcp_server_envs nor a " - f"environment_name); nothing to apply, likely a task authoring error" + f"apply_server_config: broadcast directive matched no MCP servers or websites " + f"in env {self.env_id!r} (its card lists no child envs, and its definition names neither " + f"mcp_server_envs, website_envs nor an environment_name); nothing to apply, likely a task authoring error" ) - expanded: list[ConfigDirective] = [] + expanded: list[tuple[ConfigDirective, Optional[BackingServer], bool]] = [] for d in self.directives: if d.service != self._BROADCAST: - expanded.append(d) + named = [s for s in servers if s.environment_name == d.service] + expanded.append((d, next((s for s in named if s.mcp), named[0] if named else None), False)) continue - for svc in environment_names: - expanded.append(ConfigDirective(service=svc, uri=d.uri, args=d.args)) + for server in servers: + expanded.append((ConfigDirective(service=server.environment_name, uri=d.uri, args=d.args), server, not server.mcp)) return expanded async def execute(self, context: TaskStepContext) -> TaskStepContext: @@ -180,7 +181,7 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: # actually armed with — a partial apply must not look like a no-op. changes = context.metadata.setdefault("server_config_changes", []) # Directives skipped because the target server doesn't advertise the - # extension (only when tolerate_unadvertised). Kept separate from + # extension (when tolerate_unadvertised, or for a website a broadcast reached). Kept separate from # `changes` so the audit trail never shows a skip as an applied change. skipped = context.metadata.setdefault("server_config_skipped", []) # Expand broadcast (service="*") directives to one-per-env-service. @@ -188,7 +189,7 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: # network), so wrap it to keep the broadcast path's failure contract the # same ApplyServerConfigError as the per-directive HTTP calls below. try: - directives = self._expand_directives(deployed) + directives = await self._expand_directives(deployed) except ApplyServerConfigError: raise except Exception as e: @@ -203,16 +204,19 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: # multiple tools) against the same server. A None card marks a child env with no # card: a server predating env-card support, one whose card name differs, or one # whose card the deploy couldn't read. - cards: dict[str, tuple[str, Optional[dict]]] = {} - for directive in directives: + cards: dict[Any, tuple[Optional[str], Optional[dict]]] = {} + for directive, server, lenient in directives: + tolerate = self.tolerate_unadvertised or lenient + key = directive.service if server is None else server try: - if directive.service not in cards: - cards[directive.service] = await legacy_protocol.child_env_card( - deployed, gateway_url_of(deployed), directive.service, timeout=self.timeout_seconds - ) - base_url, card = cards[directive.service] + if key not in cards: + cards[key] = (await legacy_protocol.child_env_card(deployed, gateway_url_of(deployed), directive.service, + timeout=self.timeout_seconds) + if server is None else + await legacy_protocol.server_card(server, gateway_url_of(deployed), timeout=self.timeout_seconds)) + base_url, card = cards[key] if card is None: - if not self.tolerate_unadvertised: + if not tolerate: raise ApplyServerConfigError( f"Service {directive.service!r} has no env card (env={self.env_id}), so {directive.uri!r} " f"can't be applied: it serves none, its card isn't named {directive.service!r}, or the deploy couldn't read it" @@ -220,7 +224,7 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: logger.warning( f"apply_server_config: service {directive.service!r} serves no " f"environment card (env={self.env_id}); skipping {directive.uri!r} " - f"(tolerate_unadvertised)" + f"({'tolerate_unadvertised' if self.tolerate_unadvertised else 'broadcast to a website'})" ) skipped.append({ "step_id": self.id, @@ -233,11 +237,11 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: continue ext = protocol_v1.find_extension(card, directive.uri) if ext is None: - if self.tolerate_unadvertised: + if tolerate: logger.warning( f"apply_server_config: {directive.uri!r} not advertised on " f"service {directive.service!r} (env={self.env_id}); skipping " - f"(tolerate_unadvertised)" + f"({'tolerate_unadvertised' if self.tolerate_unadvertised else 'broadcast to a website'})" ) skipped.append({ "step_id": self.id, diff --git a/src/agent_env/task_step/task_steps/deploy_agent.py b/src/agent_env/task_step/task_steps/deploy_agent.py index cbacdc804..1f37a3847 100644 --- a/src/agent_env/task_step/task_steps/deploy_agent.py +++ b/src/agent_env/task_step/task_steps/deploy_agent.py @@ -31,7 +31,7 @@ from agent_env.a2a_agent.staging import StagedObjectStore, transfer_store from agent_env.config import get_config from agent_env.env.env import DeployedEnv, DeployedSandboxEnv, Env -from agent_env.env.gateway import AGENT_ENV_ROLE_HEADER +from agent_env.env.gateway import AGENT_ENV_ROLE_HEADER, AGENT_ENV_ROLE_KEY_HEADER from agent_env.providers.sandbox_providers.local_docker.provider import host_url_for from agent_env.a2a_agent.a2a_agent import DEFAULT_A2A_PORT, unwritten_local_agent from agent_env.providers.sandbox_providers.sandbox_provider import ( @@ -70,9 +70,11 @@ def _mcp_add_request(mcp_ext: dict) -> dict: return request if isinstance(request, dict) else {} -def _mcp_add_body(mcp_ext: dict, url: str, headers: dict | None, card_name: str | None, role: str | None = None) -> dict: +def _mcp_add_body(mcp_ext: dict, url: str, headers: dict | None, card_name: str | None, role: str | None = None, + role_key: str | None = None) -> dict: """Body for the agent's mcp-config `add`. The card name and the role header ride along only when the card's `add` - takes them, judged on the exact field set sent; what an env does with the role is the env's business.""" + takes them, judged on the exact field set sent, the deploy's `role_key` beside the role so its gateway believes + it; what an env does with the role is the env's business.""" request = _mcp_add_request(mcp_ext) body: dict = {"url": url} if headers: @@ -81,6 +83,8 @@ def _mcp_add_body(mcp_ext: dict, url: str, headers: dict | None, card_name: str body["name"] = card_name if role and card_request_accepts(request, {*body, "headers"}): body.setdefault("headers", {})[AGENT_ENV_ROLE_HEADER] = role + if role_key: + body["headers"][AGENT_ENV_ROLE_KEY_HEADER] = role_key return body @@ -433,7 +437,8 @@ async def _execute(self, context: TaskStepContext, deployed_agents: list) -> Tas url = host_url_for(live_mcp_url(env_id), deployed.sandbox_type) card_name = None if mcp_ext: - body = _mcp_add_body(mcp_ext, url, sandbox_request_headers_for_url(url), card_name, self.role) + body = _mcp_add_body(mcp_ext, url, sandbox_request_headers_for_url(url), card_name, self.role, + deployed_env.role_key if deployed_env is not None else None) registrations.append((env_id, body)) identity_payload: dict[str, str] = {"name": self.agent_name} diff --git a/src/agent_env/task_step/task_steps/deploy_env.py b/src/agent_env/task_step/task_steps/deploy_env.py index 375a4c515..d12609f75 100644 --- a/src/agent_env/task_step/task_steps/deploy_env.py +++ b/src/agent_env/task_step/task_steps/deploy_env.py @@ -48,6 +48,8 @@ def __init__( artifact_id: Optional[str] = None, artifact_version: Optional[int] = None, env_params: Optional[dict] = None, + keyed_roles: bool = False, + capture_traffic: bool = False, ): super().__init__(id, version, depends_on=depends_on, fail_task_on_error=fail_task_on_error) self.env_id = env_id @@ -66,6 +68,8 @@ def __init__( self.metadata = metadata or {} # The env type's own deploy options, which this step passes to its deploy() without reading them. self.env_params = env_params + self.keyed_roles = keyed_roles + self.capture_traffic = capture_traffic def run_state(self, user_overrides: Mapping[str, Any]) -> tuple[str | None, str | None]: """The env state type and attached state instance a run with ``user_overrides`` deploys on: each the run's, @@ -89,6 +93,9 @@ def to_dict(self) -> dict: base["env_state_instance_id"] = self.env_state_instance_id base["metadata"] = self.metadata base["env_params"] = self.env_params + base["keyed_roles"] = self.keyed_roles + if self.capture_traffic: + base["capture_traffic"] = True return base @classmethod @@ -109,6 +116,8 @@ def from_dict(cls, data: dict) -> DeployEnvTaskStep: env_state_instance_id=data.get("env_state_instance_id"), metadata=dict(data.get("metadata") or {}), env_params=data.get("env_params"), + keyed_roles=bool(data.get("keyed_roles", False)), + capture_traffic=bool(data.get("capture_traffic", False)), ) def preflight(self) -> list[str]: @@ -133,7 +142,9 @@ def preflight(self) -> list[str]: return [f"deploy_env '{self.id}': {refusal}"] if refusal else [] def _env_params_option(self) -> dict: - return {"env_params": self.env_params} if self.env_params else {} + return {**({"env_params": self.env_params} if self.env_params else {}), + **({"keyed_roles": True} if self.keyed_roles else {}), + **({"capture_traffic": True} if self.capture_traffic else {})} async def execute(self, context: TaskStepContext) -> TaskStepContext: from agent_env.env.env import Env @@ -187,11 +198,21 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: if self.env_params: if "env_params" not in params and not accepts_kwargs: raise ValueError(f"deploy_env '{self.id}': {env.type} envs take no env_params") - deploy_kwargs["env_params"] = self.env_params + deploy_kwargs["env_params"] = await env.run_env_params(self.env_params, context.deployed_envs) + if self.keyed_roles: + if "keyed_roles" not in params and not accepts_kwargs: + raise ValueError(f"deploy_env '{self.id}': {env.type} envs take no keyed_roles") + deploy_kwargs["keyed_roles"] = True + if self.capture_traffic: + if "capture_traffic" not in params and not accepts_kwargs: + raise ValueError(f"deploy_env '{self.id}': {env.type} envs take no capture_traffic") + deploy_kwargs["capture_traffic"] = True deployed_env = await env.deploy(**deploy_kwargs) # create_instance runs inside env.deploy(), so an annotation set after it # returns has to be persisted explicitly to reach the instance record. - annotations = {"deploy_step_id": self.id} + from agent_env.task_step.task_steps.sign_in_browser import mark_live + + annotations = {"deploy_step_id": self.id, **mark_live(context, deployed_env)} deployed_env.metadata = {**(deployed_env.metadata or {}), **annotations} if deployed_env.instance_id: update_env_instance_metadata(deployed_env.instance_id, annotations) diff --git a/src/agent_env/task_step/task_steps/load_artifact.py b/src/agent_env/task_step/task_steps/load_artifact.py index fdecb9e75..c79d6153c 100644 --- a/src/agent_env/task_step/task_steps/load_artifact.py +++ b/src/agent_env/task_step/task_steps/load_artifact.py @@ -645,7 +645,8 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: if not (agent.on_host and agent.sandbox_type == LocalSandbox.type): gateway_url = reachable_url(gateway_url, from_sandbox_type=deployed.sandbox_type, to_sandbox_type=agent.sandbox_type) - install_path = await A2AAgent.install_cli(deployed_agent, artifact, gateway_url, **_agent_files(agent)) + install_path = await A2AAgent.install_cli(deployed_agent, artifact, gateway_url, role_key=deployed.role_key, + **_agent_files(agent)) agent_clis = context.metadata.setdefault("installed_clis", {}).setdefault(self.agent_name, {}) agent_clis[artifact.id] = { "install_path": install_path, diff --git a/src/agent_env/task_step/task_steps/mcp_cli_builder/build_mcp_cli.py b/src/agent_env/task_step/task_steps/mcp_cli_builder/build_mcp_cli.py index e82ba5b0d..62eb55fb6 100644 --- a/src/agent_env/task_step/task_steps/mcp_cli_builder/build_mcp_cli.py +++ b/src/agent_env/task_step/task_steps/mcp_cli_builder/build_mcp_cli.py @@ -17,7 +17,7 @@ from agent_env.artifact import CliArtifact from agent_env.artifact.store import artifact_write_lock from agent_env.env import Env -from agent_env.env.env import require_gateway_url +from agent_env.env.env import require_gateway_url, website_child from agent_env.store.ids import derive_id from agent_env.task_step.context import TaskStepContext from agent_env.entity.refs import EntityRef, RefRole @@ -40,11 +40,12 @@ async def _resolve_manifest_endpoint(gateway_url: str, environment_name: str) -> logger.debug("env card for %r unavailable (%s); constructing the manifest path", environment_name, e) return None - child = protocol_v1.find_child(card, environment_name) - if child is None: - logger.debug("env card has no child named %r; constructing the manifest path", environment_name) + found = [child for child in protocol_v1.children(card) if child.name == environment_name and not website_child(child)] + if len(found) != 1: + logger.debug("env card has %s MCP server child named %r; constructing the manifest path", + len(found) or "no", environment_name) return None - return protocol_v1.extension_params(child, GET_INTERFACES_EXTENSION_URI).get("endpoint") + return protocol_v1.extension_params(found[0].card, GET_INTERFACES_EXTENSION_URI).get("endpoint") async def fetch_interface_manifest( diff --git a/src/agent_env/task_step/task_steps/mcp_cli_builder/codegen.py b/src/agent_env/task_step/task_steps/mcp_cli_builder/codegen.py index 4be0384b0..9853400f2 100644 --- a/src/agent_env/task_step/task_steps/mcp_cli_builder/codegen.py +++ b/src/agent_env/task_step/task_steps/mcp_cli_builder/codegen.py @@ -14,10 +14,11 @@ from agentenv_protocol import MANIFEST_VERSION, manifest_compatible -from agent_env.env.gateway import AGENT_ENV_ROLE_HEADER +from agent_env.env.gateway import AGENT_ENV_ROLE_HEADER, AGENT_ENV_ROLE_KEY_HEADER GATEWAY_URL_ENV_VAR = "AGENT_ENV_GATEWAY_URL" ROLE_ENV_VAR = "AGENT_ENV_ROLE" +ROLE_KEY_ENV_VAR = "AGENT_ENV_ROLE_KEY" DEFAULT_CLI_ROLE = "cli" _COMMAND_NAME_RE = re.compile(r"^[a-zA-Z][a-zA-Z0-9_-]*$") _WORD_RE = re.compile(r"[A-Z]+(?=[A-Z][a-z]|$)|[A-Z]?[a-z]+|[0-9]+") @@ -166,6 +167,8 @@ def generate_cli_script( role_env_var=ROLE_ENV_VAR, default_role=DEFAULT_CLI_ROLE, role_header=AGENT_ENV_ROLE_HEADER, + role_key_env_var=ROLE_KEY_ENV_VAR, + role_key_header=AGENT_ENV_ROLE_KEY_HEADER, ) @@ -186,6 +189,8 @@ def generate_cli_script( ROLE_ENV = "{role_env_var}" DEFAULT_ROLE = "{default_role}" ROLE_HEADER = "{role_header}" +ROLE_KEY_ENV = "{role_key_env_var}" +ROLE_KEY_HEADER = "{role_key_header}" SERVICE_NAME = {environment_name} INTERFACE_MANIFEST = {interface_manifest_literal} _CONFIG_FILE_NAME = ".env" @@ -226,11 +231,17 @@ def _role(): return os.environ.get(ROLE_ENV, DEFAULT_ROLE) +def _role_headers(): + """The role header, with the deploy's role key beside it when one is set, so the gateway believes the role.""" + key = os.environ.get(ROLE_KEY_ENV) + return {{ROLE_HEADER: _role(), **({{ROLE_KEY_HEADER: key}} if key else {{}})}} + + def _post_step(payload): response = httpx.post( f"{{_gateway_url()}}/step", json=payload, - headers={{ROLE_HEADER: _role()}}, + headers=_role_headers(), timeout=120, ) response.raise_for_status() @@ -248,7 +259,7 @@ def _list_tools(): allowed_names = {{t.get("name") for t in (role_filtered.get("tools") or []) if t.get("name")}} state_resp = httpx.get( f"{{_gateway_url()}}/state", - headers={{ROLE_HEADER: _role()}}, + headers=_role_headers(), timeout=120, ) state_resp.raise_for_status() diff --git a/src/agent_env/task_step/task_steps/post_turn_hooks.py b/src/agent_env/task_step/task_steps/post_turn_hooks.py index 0afb54b71..cca67b1fd 100644 --- a/src/agent_env/task_step/task_steps/post_turn_hooks.py +++ b/src/agent_env/task_step/task_steps/post_turn_hooks.py @@ -22,7 +22,6 @@ from agent_env.artifact import FileArtifact from agent_env.env.env import DeployedEnv, EnvCapabilityUnsupported from agent_env.env.gateway.constants import ( - AGENT_ENV_ROLE_HEADER, EXT_CLOCK_URI, EXT_NOTIFICATIONS_URI, EXT_STEP_URI, @@ -152,7 +151,7 @@ async def call_tool(self, tool: str, args: Optional[dict], role: str) -> dict: "source": STEP_SOURCE_POST_TURN_HOOK, "hook_artifact_id": self._hook_artifact_id} async with httpx.AsyncClient() as client: resp = await client.request(op["method"], f"{self._deployed.environment_url}{op['endpoint']}", json=body, - headers={AGENT_ENV_ROLE_HEADER: role, **self._deployed.control_headers}, + headers={**self._deployed.role_headers(role), **self._deployed.control_headers}, timeout=_TOOL_CALL_TIMEOUT_S) if resp.status_code >= 400: raise RuntimeError(f"env '{self.env_id}': /step call_tool {tool!r} failed (HTTP {resp.status_code}): " diff --git a/src/agent_env/task_step/task_steps/register_env_triggers.py b/src/agent_env/task_step/task_steps/register_env_triggers.py index 156d17fb9..237f77259 100644 --- a/src/agent_env/task_step/task_steps/register_env_triggers.py +++ b/src/agent_env/task_step/task_steps/register_env_triggers.py @@ -21,6 +21,7 @@ TRIGGER_PROVOKING_ROLES, TRIGGER_UNLESS, TRIGGER_WEBSITE, + TRIGGER_WEBSITE_OP, ) from agent_env.task_step.context import TaskStepContext from agent_env.entity.refs import EntityRef @@ -146,12 +147,15 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: f"redeploy it on a gateway image that does.") # An older gateway ignores what it does not declare: named actions would go to the default executor, a # trigger's when.roles would leave it to the watched roles, delays and hours would not wait, a deadline - # would fire even after its `unless` call, and hours in a time zone would be read as UTC. + # would fire even after its `unless` call, hours in a time zone would be read as UTC, and a website trigger + # naming an `op` would fire on every request to the site. used = {TRIGGER_NAMED_EXECUTORS: bool(named), TRIGGER_PROVOKING_ROLES: _when_has(self.triggers, "roles"), TRIGGER_ACTION_DELAY: _delays(self.triggers), TRIGGER_EXECUTOR_HOURS: bool(self.executor_hours), TRIGGER_UNLESS: _when_has(self.triggers, "unless"), TRIGGER_WEBSITE: any(isinstance(t.get("when"), dict) and t["when"].get("type") == "website" for t in self.triggers), + TRIGGER_WEBSITE_OP: any(isinstance(t.get("when"), dict) and t["when"].get("type") == "website" + and t["when"].get("op") is not None for t in self.triggers), TRIGGER_EXECUTOR_HOURS_TIME_ZONE: _hours_in_a_time_zone(self.executor_hours), TRIGGER_NOTIFY: any(a.get("type") == "notify" for t in self.triggers for a in t.get("actions") or [] if isinstance(a, dict))} diff --git a/src/agent_env/task_step/task_steps/reset_env.py b/src/agent_env/task_step/task_steps/reset_env.py index 88059fdde..59d1717d1 100644 --- a/src/agent_env/task_step/task_steps/reset_env.py +++ b/src/agent_env/task_step/task_steps/reset_env.py @@ -3,7 +3,8 @@ Env-agnostic: looks up the deployed env in context and calls its ``reset()``. Best-effort by default — a reset failure (including an env that doesn't support it) is logged and the run continues. Place right before prompt_agent in a -warm-reuse task. +warm-reuse task. After a reset, the browser is signed in again to the live websites +a sign_in_browser step of the run signed it in to. """ from __future__ import annotations @@ -61,6 +62,9 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: env = Env.get(deployed.env_id, deployed.env_version) await env.reset(deployed) logger.info("reset_env: reset env '%s'", self.env_id) + from agent_env.task_step.task_steps.sign_in_browser import sign_in_again + + await sign_in_again(context, deployed) except NotImplementedError: # Env doesn't support reset — not an error. logger.info("reset_env: env '%s' does not support reset; skipping", self.env_id) diff --git a/src/agent_env/task_step/task_steps/sign_in_browser.py b/src/agent_env/task_step/task_steps/sign_in_browser.py new file mode 100644 index 000000000..2af309270 --- /dev/null +++ b/src/agent_env/task_step/task_steps/sign_in_browser.py @@ -0,0 +1,316 @@ +"""Live websites in a run: marking a run that uses one, and signing the browser in to them. + +A website child env whose card's website/v1 says ``live`` is a real site. ``mark_live``, which deploy_env runs on every +env it deploys, records each one under ``context.metadata["live_websites"]``, sets ``context.metadata["reproducible"]`` +to False, and marks the deploy record ``live``. It also checks the site's domains and ``allow_hosts`` against the network +allowlist of each computer deployed in the run. An allowlist is fixed when its sandbox starts, so a host one leaves out +is recorded under ``unreachable`` and warned about, not added; a run whose sandboxes restrict no egress records +``network: "unrestricted"``, since nothing there enforces ``allow_hosts`` either. + +``sign_in_browser`` calls each live site's website/v1 ``session`` and hands the cookies straight to the browser's +browser/v1 ``cookies/set``, each called on the child env's own card as the env's card nests it, so a site sharing its +name with an MCP server, or with another child, is still the one called. The cookies never enter the context, a log or +the step's record, which keeps only each cookie's name, domain, path and expiry. reset_env signs the browser in again after a reset, as the last sign-in did. +With several deployments of the env in a run, ``env_step_id`` names the deploy_env step whose deployment it signs in; the +record keeps that step and the call timeout, so the sign-in after a reset reaches the same browser the same way. +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass, field +from typing import Any, ClassVar, Optional + +import httpx +from agentenv_protocol import client as protocol_v1 +from agentenv_protocol.website import BROWSER_EXTENSION_URI, CLEAR_COOKIES, SET_COOKIES, WEBSITE_EXTENSION_URI +from agentenv_protocol.website.browser import cookies as parse_cookies +from agentenv_protocol.website.browser import redacted + +from agent_env.entity.refs import EntityRef +from agent_env.env.env import EnvCapabilityUnsupported +from agent_env.task_step.context import TaskStepContext +from agent_env.task_step.task_step import TaskStep, TaskStepDependency + +logger = logging.getLogger(__name__) + +LIVE_KEY = "live_websites" +REPRODUCIBLE_KEY = "reproducible" +SIGN_INS_KEY = "browser_sign_ins" +_ERROR_CHARS = 500 +SIGN_IN_TIMEOUT_SECONDS = 30 + + +@dataclass(frozen=True) +class LiveSite: + """A live website child env, as its card declares it; ``child`` is that child env, which its session is called on, + and None for a site read back from a record.""" + + name: str + domains: tuple[str, ...] + allow_hosts: tuple[str, ...] + signs_in: bool + child: Optional[protocol_v1.ChildEnv] = field(default=None, compare=False, repr=False) + + def hosts(self) -> list[str]: + return [*self.domains, *(h for h in self.allow_hosts if h not in self.domains)] + + def record(self) -> dict: + return {"name": self.name, "domains": list(self.domains), "allow_hosts": list(self.allow_hosts), + "signs_in": self.signs_in} + + +def _children(card: dict) -> list[protocol_v1.ChildEnv]: + """``card``'s child envs, or ``card`` itself, at the env's own address, for a leaf card.""" + if isinstance(card.get("children_environments"), list): + return protocol_v1.children(card) + return [protocol_v1.ChildEnv(name=card.get("name"), url=None, card=card)] + + +def live_sites(card: Optional[dict]) -> list[LiveSite]: + """The live websites among ``card``'s child envs, or ``card`` itself for a leaf card.""" + sites = [] + for child in _children(card or {}): + params = (child.find_extension(WEBSITE_EXTENSION_URI) or {}).get("params") or {} + if params.get("live") is True and isinstance(child.name, str): + sites.append(LiveSite(child.name, tuple(params.get("domains") or ()), + tuple(params.get("allow_hosts") or ()), + params.get("signs_in") is not False + and protocol_v1.find_extension_method(child.card, WEBSITE_EXTENSION_URI, + "session") is not None, child)) + return sites + + +def browsers(card: Optional[dict]) -> list[protocol_v1.ChildEnv]: + """``card``'s child envs that serve browser/v1's cookies/set.""" + return [child for child in _children(card or {}) if isinstance(child.name, str) + and protocol_v1.find_extension_method(child.card, BROWSER_EXTENSION_URI, SET_COOKIES) is not None] + + +async def _invoke(deployed: Any, child: protocol_v1.ChildEnv, uri: str, method: str, params: dict, + timeout: int) -> Any: + """Call ``method`` of the extension at ``uri`` as ``child``'s own card advertises it, at ``deployed``'s address + with its control headers, which a lookup of the child by its name could confuse with another of that name.""" + if not deployed.environment_url or protocol_v1.find_extension_method(child.card, uri, method) is None: + raise EnvCapabilityUnsupported(uri, method, deployed.env_id, deployed.instance_id, child.name) + return await protocol_v1.invoke_extension(deployed.environment_url, child.card, uri, params, timeout, + method=method, headers=deployed.control_headers) + + +def _allows(entries: list[str], host: str) -> bool: + host = host.lower().rstrip(".") + for entry in (e.lower().rstrip(".") for e in entries): + if entry == host or (entry.startswith("*.") and host.endswith(entry[1:])): + return True + return False + + +def network_check(context: TaskStepContext, sites: list[LiveSite], deployed: Any = None) -> dict: + """How the run's computers' network allowlists treat the live ``sites``' hosts: each restricting computer's + hosts it leaves out, or ``unrestricted`` when no computer in the run restricts egress. ``deployed``, an env being + deployed that the context doesn't hold yet, is one of the run's.""" + from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy + + hosts = list(dict.fromkeys(h for site in sites for h in site.hosts())) + unreachable: dict[str, list[str]] = {} + restricted = False + envs = [*context.deployed_envs, *([deployed] if deployed is not None and deployed not in context.deployed_envs + else [])] + for env in envs: + network = ((env.metadata or {}).get("computer") or {}).get("network") + if not network: + continue + policy = NetworkPolicy.from_dict(network) + if not policy.restricts_egress: + continue + restricted = True + if missing := [h for h in hosts if not _allows(list(policy.allow_hosts), h)]: + unreachable[env.env_id] = missing + if not restricted: + return {"network": "unrestricted"} + return {"network": "allowlist", "unreachable": unreachable} + + +def mark_live(context: TaskStepContext, deployed: Any) -> dict: + """Record ``deployed``'s live websites in ``context`` and return what its deploy record gains: ``live`` and + ``reproducible`` when it has one, nothing when it has none. The network check of every live website recorded so + far is made again with ``deployed`` among the run's computers, so a computer deployed after a site is checked + too.""" + sites = live_sites(deployed.environment_card) + if sites: + context.metadata.setdefault(LIVE_KEY, {})[deployed.env_id] = {"sites": [s.record() for s in sites]} + context.metadata[REPRODUCIBLE_KEY] = False + for env_id, entry in (context.metadata.get(LIVE_KEY) or {}).items(): + before = entry.pop("unreachable", None) or {} + entry_sites = sites if env_id == deployed.env_id else [ + LiveSite(r["name"], tuple(r["domains"]), tuple(r["allow_hosts"]), r["signs_in"]) for r in entry["sites"]] + network = network_check(context, entry_sites, deployed) + entry.update(network) + for computer, missing in (network.get("unreachable") or {}).items(): + if before.get(computer) != missing: + logger.warning("live websites of env %s: env %s's network allowlist leaves out %s, which it can't add " + "once its sandbox runs, so those pages won't load there", env_id, computer, missing) + if env_id == deployed.env_id and network["network"] == "unrestricted": + logger.info("live websites of env %s (%s): no sandbox in this run restricts egress, so allow_hosts is " + "recorded, not enforced", env_id, [s.name for s in sites]) + return {"live": True, REPRODUCIBLE_KEY: False} if sites else {} + + +async def sign_in(context: TaskStepContext, deployed: Any, *, users: Optional[dict[str, str]] = None, + sites: Optional[list[str]] = None, browser: Optional[str] = None, clear: bool = True, + timeout: int = SIGN_IN_TIMEOUT_SECONDS) -> dict: + """Sign ``deployed``'s browser in to its live websites (those named in ``sites``, else every one with a session), + each as ``users[name]`` or the site's own choice; with ``clear``, a site's old cookies go first, on a browser that + serves cookies/clear. Returns the record of what was set, cookies by everything but their values.""" + card = deployed.environment_card or {} + live = {site.name: site for site in live_sites(card)} + unknown = sorted(set(sites or []) - set(live)) + if unknown: + raise RuntimeError(f"env {deployed.env_id!r} has no live website named {unknown}; its live ones are " + f"{sorted(live) or 'none'}") + chosen = [live[name] for name in sites] if sites else [s for s in live.values() if s.signs_in] + offered = [child for child in browsers(card) if browser is None or child.name == browser] + if chosen and not offered: + named = f"named {browser!r} " if browser is not None else "" + raise RuntimeError(f"env {deployed.env_id!r} has live websites with a session ({[s.name for s in chosen]}) " + f"but no child env {named}serves {BROWSER_EXTENSION_URI} {SET_COOKIES} to sign a browser in") + if chosen and browser is not None and len(offered) > 1: + raise RuntimeError(f"env {deployed.env_id!r} has {len(offered)} browsers named {browser!r} " + f"({', '.join(str(c.url) for c in offered)}), so the name can't say which to sign in") + target = offered[0] if chosen else None + browser = target.name if target is not None else browser + record: dict = {"env_id": deployed.env_id, "env_step_id": deploy_step_of(deployed), "browser": browser, + "timeout_seconds": timeout, "sites": []} + for site in chosen: + user = (users or {}).get(site.name) + try: + session = await _invoke(deployed, site.child, WEBSITE_EXTENSION_URI, "session", {"user": user}, timeout) + except Exception as e: + raise RuntimeError(f"live website {site.name!r}: its session failed ({_why(e)})") from None + try: + given = parse_cookies(session.get("cookies") if isinstance(session, dict) else None, + default_domain=site.domains[0] if site.domains else None) + except ValueError as e: + raise RuntimeError(f"live website {site.name!r}: its session returned {e}") from None + try: + if clear and site.domains and protocol_v1.find_extension_method(target.card, BROWSER_EXTENSION_URI, + CLEAR_COOKIES) is not None: + await _invoke(deployed, target, BROWSER_EXTENSION_URI, CLEAR_COOKIES, {"domains": list(site.domains)}, + timeout) + await _invoke(deployed, target, BROWSER_EXTENSION_URI, SET_COOKIES, + {"cookies": [c.model_dump(exclude_none=True) for c in given]}, timeout) + except Exception as e: + raise RuntimeError(f"browser {browser!r}: setting {site.name!r}'s cookies failed ({_why(e)})") from None + record["sites"].append({"name": site.name, "user": user, "expires_at": session.get("expires_at"), + "cookies": redacted(given)}) + logger.info("sign_in_browser: %s signed in to %s as %s (%s)", browser, site.name, user or "its own choice", + [c.name for c in given]) + return record + + +def _why(error: Exception) -> str: + """An error as a record keeps it: a call's status and error code, or the error's kind, never a message that may + repeat a cookie.""" + if isinstance(error, httpx.HTTPStatusError): + code = getattr(error, "code", None) + return f"HTTP {error.response.status_code}{f' {code}' if code else ''}" + if isinstance(error, EnvCapabilityUnsupported): + return str(error) + return type(error).__name__ + + +def deploy_step_of(deployed: Any) -> Optional[str]: + """The id of the deploy_env step that deployed ``deployed``, or None.""" + return (getattr(deployed, "metadata", None) or {}).get("deploy_step_id") + + +def deployment(context: TaskStepContext, env_id: str, env_step_id: Optional[str], step_id: str) -> Any: + """The deployment of ``env_id`` a sign-in reaches: the one deploy_env step ``env_step_id`` made, else the only one; + raises when there is none, or several and no ``env_step_id`` says which.""" + matches = [d for d in context.deployed_envs if d.env_id == env_id] + if env_step_id is not None: + named = [d for d in matches if deploy_step_of(d) == env_step_id] + if not named: + raise RuntimeError(f"sign_in_browser {step_id!r}: env_step_id {env_step_id!r} names no deployment of env " + f"{env_id!r} (deployed by: {[deploy_step_of(d) for d in matches]})") + matches = named + if not matches: + raise RuntimeError(f"Env '{env_id}' not found in context.deployed_envs") + if len(matches) > 1: + raise RuntimeError(f"sign_in_browser {step_id!r}: {len(matches)} deployments of env {env_id!r} (deployed by: " + f"{[deploy_step_of(d) for d in matches]}); set env_step_id to the deploy_env step whose " + "browser to sign in") + return matches[0] + + +async def sign_in_again(context: TaskStepContext, deployed: Any) -> None: + """After a reset of ``deployed``, sign its browser in again as each sign-in of this run that reached it did, with + the same timeout; never raises.""" + for step_id, entry in list((context.metadata.get(SIGN_INS_KEY) or {}).items()): + if entry.get("env_id") != deployed.env_id or not entry.get("sites"): + continue + if entry.get("env_step_id") is not None and entry["env_step_id"] != deploy_step_of(deployed): + continue + users = {s["name"]: s["user"] for s in entry["sites"] if s.get("user")} + try: + context.metadata[SIGN_INS_KEY][step_id] = await sign_in( + context, deployed, users=users, sites=[s["name"] for s in entry["sites"]], browser=entry.get("browser"), + timeout=entry.get("timeout_seconds") or SIGN_IN_TIMEOUT_SECONDS) + except Exception as e: + logger.warning("reset_env: signing env %s's browser in again failed: %s", deployed.env_id, e) + entry["error"] = str(e)[:_ERROR_CHARS] + + +class SignInBrowserTaskStep(TaskStep): + """Sign a deployed env's browser in to its live websites before the agent starts. ``users`` maps a site's name to + the user to sign in as; ``sites`` limits it to some live sites; ``browser`` names the child env serving browser/v1 + when there are several; ``env_step_id`` names the deploy_env step whose deployment it signs in, when the env is + deployed more than once.""" + + type: ClassVar[str] = "sign_in_browser" + entity_refs = (EntityRef.env("env_id"),) + + def __init__(self, id: str, version: Optional[int], env_id: str, users: Optional[dict[str, str]] = None, + sites: Optional[list[str]] = None, browser: Optional[str] = None, + depends_on: Optional[list[TaskStepDependency]] = None, fail_task_on_error: bool = True, + timeout_seconds: int = SIGN_IN_TIMEOUT_SECONDS, env_step_id: Optional[str] = None): + super().__init__(id, version, depends_on=depends_on, fail_task_on_error=fail_task_on_error) + if not env_id: + raise ValueError("sign_in_browser needs the env_id of the env whose browser it signs in") + if users is not None and not (isinstance(users, dict) and all( + isinstance(k, str) and isinstance(v, str) for k, v in users.items())): + raise ValueError("sign_in_browser's users maps a site's name to a user's") + self.env_id, self.users, self.sites, self.browser = env_id, users, sites, browser + self.timeout_seconds, self.env_step_id = timeout_seconds, env_step_id + + def to_dict(self) -> dict: + data = {**super().to_dict(), "env_id": self.env_id, "timeout_seconds": self.timeout_seconds} + for key in ("users", "sites", "browser", "env_step_id"): + if getattr(self, key) is not None: + data[key] = getattr(self, key) + return data + + @classmethod + def from_dict(cls, data: dict) -> SignInBrowserTaskStep: + return cls(**cls._base_from_dict(data), env_id=data["env_id"], users=data.get("users"), + sites=data.get("sites"), browser=data.get("browser"), + timeout_seconds=data.get("timeout_seconds", SIGN_IN_TIMEOUT_SECONDS), env_step_id=data.get("env_step_id")) + + async def execute(self, context: TaskStepContext) -> TaskStepContext: + deployed = deployment(context, self.env_id, self.env_step_id, self.id) + if live_sites(deployed.environment_card) and self.env_id not in (context.metadata.get(LIVE_KEY) or {}): + mark_live(context, deployed) + try: + record = await sign_in(context, deployed, users=self.users, sites=self.sites, browser=self.browser, + timeout=self.timeout_seconds) + except Exception as e: + context.metadata.setdefault(SIGN_INS_KEY, {})[self.id] = {"env_id": self.env_id, "error": str(e)[:_ERROR_CHARS]} + logger.warning("sign_in_browser %s: %s", self.id, e) + if self.fail_task_on_error: + raise + return context + context.metadata.setdefault(SIGN_INS_KEY, {})[self.id] = record + if not record["sites"]: + logger.info("sign_in_browser %s: env %s has no live website with a session; nothing to sign in to", + self.id, self.env_id) + return context diff --git a/src/agent_env/task_step/task_steps/snapshot_env.py b/src/agent_env/task_step/task_steps/snapshot_env.py index 48fc9d6dc..abb5573f3 100644 --- a/src/agent_env/task_step/task_steps/snapshot_env.py +++ b/src/agent_env/task_step/task_steps/snapshot_env.py @@ -13,7 +13,7 @@ from dataclasses import dataclass from datetime import datetime, timezone from pathlib import PurePosixPath -from typing import ClassVar, Optional +from typing import TYPE_CHECKING, ClassVar, Optional from urllib.parse import urlparse import httpx @@ -37,6 +37,9 @@ from agent_env.entity.scopes import FieldScope from agent_env.task_step.task_step import TaskStep, TaskStepDependency +if TYPE_CHECKING: + from agent_env.env.legacy_protocol import BackingServer + logger = logging.getLogger(__name__) DEFAULT_EXPORT_TIMEOUT_SECONDS = 600 @@ -167,6 +170,9 @@ class SnapshotEnvTaskStep(TaskStep): Output: ``context.metadata["env_snapshotted_universes"][self.id]`` = ``{"id", "version"}``. Fails the step if any service fails to export. + Websites are snapshotted through their v1 data plane; a website serving + none is left out, and one sharing its name with an MCP server is left to + the MCP server, as a MultiEnv loads that name's artifact into it. ``include_env_changes`` also stores who changed what (urn:agentenv:changes/v1) under ``metadata["env_changes"]``. @@ -470,27 +476,42 @@ def _resolve_loaded_universe_ref(deployed) -> Optional[tuple[str, Optional[int]] return (su["id"], su.get("version")) if su and su.get("id") else None @staticmethod - def _enumerate_environments(env) -> list[str]: - """environment names for a multi env or a single MCP server env.""" - if getattr(env, "mcp_server_envs", None): - return [e.environment_name for e in env.mcp_server_envs] - if getattr(env, "environment_name", None): - return [env.environment_name] - raise RuntimeError( - f"Env '{env.id}' v{env.version} has no MCP services to snapshot " - "(expected mcp_server_envs or environment_name)" - ) + def _snapshot_servers(servers: list[BackingServer], env) -> list[BackingServer]: + """The servers to snapshot of ``servers`` (``legacy_protocol.backing_servers``), one per environment_name: a + universe names each environment by it, and a load hands an environment of a name an MCP server and a website + share to the MCP server, so a server whose name one kept earlier holds (the env itself, else an MCP server) is + left out, and logged.""" + if not servers: + raise RuntimeError( + f"Env '{env.id}' v{env.version} has no MCP services or websites to snapshot " + "(its card lists no child envs, and its definition names no mcp_server_envs, website_envs or environment_name)" + ) + if not all(server.environment_name for server in servers): + raise RuntimeError(f"Env '{env.id}' v{env.version} has a card naming no environment to snapshot") + kept: dict[str, BackingServer] = {} + for server in sorted(servers, key=lambda s: not s.mcp): + if (holder := kept.get(server.environment_name)) is not None: + logger.warning(f"snapshot_env: {'website' if not server.mcp else 'server'} {server.environment_name!r} " + f"of {env.id} is not snapshotted: it shares its name with the {'MCP server' if holder.mcp else 'website'} " + "kept, and a universe names an environment by its name") + continue + kept[server.environment_name] = server + return list(kept.values()) @staticmethod async def _export_environment_to_file( gateway_url: Optional[str], environment_name: str, tmp_path: str, timeout_seconds: float, deployed=None, - data_get_params: Optional[dict] = None, - ) -> str | tuple[GetDataResponse, Optional[_SnapshotUpload]]: - """Export one service; return the suffix written (".json"/".zip") to - ``tmp_path``, or the object url a FilePart names when the service uploaded the - bundle itself (nothing written to ``tmp_path`` then — export_one registers it), or, for - an environment whose card declares a file tree, its data/get answer and the upload it - answered under, which export_one keeps file by file. ``data_get_params`` go to its data/get. + data_get_params: Optional[dict] = None, mcp: bool = True, server: Optional[BackingServer] = None, + ) -> Optional[str | tuple[GetDataResponse, Optional[_SnapshotUpload]]]: + """Export one service, an MCP server or with ``mcp=False`` a website; return the + suffix written (".json"/".zip") to ``tmp_path``, or the object url a FilePart names + when the service uploaded the bundle itself (nothing written to ``tmp_path`` then — + export_one registers it), or, for an environment whose card declares a file tree, its + data/get answer and the upload it answered under, which export_one keeps file by file, + or None for a service serving no v1 data plane: a website with no card, or a service whose card declares + operations without data/get. ``data_get_params`` go to its data/get. ``server`` is the service as the env's + card lists it (``legacy_protocol.backing_servers``), its card and route; without it, the service is looked up + by name on ``deployed``'s stored card, or probed live at its gateway path. v1 ``get_data``, handed a namespace grant when the service takes the data-objects form: an upload through it → its object url; ``DataPart`` → json; ``FilePart`` → @@ -499,10 +520,22 @@ async def _export_environment_to_file( guard. Errors are caught by the caller (fails just this service).""" from agent_env.env import legacy_protocol - base_url = await legacy_protocol.v1_base_url(deployed, gateway_url, environment_name, mcp=True) + if server is not None: + mcp = server.mcp + base_url = await legacy_protocol.server_v1_base_url(server, gateway_url) + else: + base_url = await legacy_protocol.v1_base_url(deployed, gateway_url, environment_name, mcp=mcp) + card_base, card = None, None + if base_url is not None: + card_base, card = (await legacy_protocol.server_card(server, gateway_url) if server is not None else + await legacy_protocol.child_env_card(deployed, gateway_url, environment_name, mcp=mcp)) + if card is not None and protocol_v1.serves(card, METHOD_GET) is False: + logger.info(f"snapshot_env: {environment_name}'s card declares {protocol_v1.operations(card)}, no {METHOD_GET}") + return None + if base_url is None and not mcp: + return None if base_url is not None: headers = deployed.control_headers_for(base_url) if deployed is not None else None - card_base, card = await legacy_protocol.child_env_card(deployed, gateway_url, environment_name) if returns_file_tree(card or {}): upload = await asyncio.to_thread(_snapshot_upload, card or {}, timeout_seconds, getattr(deployed, "sandbox_type", None), FILE_TREE_LIMITS) @@ -609,10 +642,12 @@ async def snapshot_env_state( A classmethod so callers that have no step instance can reuse it.""" from agent_env.artifact import FileArtifact, EnvironmentArtifact, EnvironmentUniverseArtifact + from agent_env.env import legacy_protocol from agent_env.env.env import gateway_url_of - services = cls._enumerate_environments(env) if gateway_url else _card_environments(deployed) gateway = gateway_url.rstrip("/") if gateway_url else None + servers = cls._snapshot_servers(await legacy_protocol.backing_servers(deployed, gateway, env), env) + services = [server.environment_name for server in servers] # `gateway` identifies which deployment was exported: env= alone doesn't when one env_id has several. logger.info( f"snapshot_env: env={env.id} v{env.version} snapshot_id={snapshot_id} " @@ -643,9 +678,12 @@ async def snapshot_env_state( # The stored card describes the deployment's own gateway; an override pointing elsewhere keeps the live probe. card_record = deployed if deployed is not None and (gateway is None or (gateway_url_of(deployed) or "").rstrip("/") == gateway) else None sem = asyncio.Semaphore(concurrency) + no_data_plane: set[str] = set() - async def export_one(environment_name: str): - """Returns EnvironmentArtifact on success, (public_err, log_err) on failure.""" + async def export_one(server: BackingServer): + """Returns EnvironmentArtifact on success, (public_err, log_err) on failure, None for a service serving no data plane.""" + environment_name = server.environment_name + website = {} if server.mcp else {"mcp": False} async with sem: # Suffix (.json/.zip) is reported by _export_environment_to_file. fd, tmp_path = tempfile.mkstemp(prefix=f"{environment_name}-") @@ -657,6 +695,7 @@ async def export_one(environment_name: str): result = await cls._export_environment_to_file( gateway, environment_name, tmp_path, export_timeout_seconds, deployed=card_record, data_get_params=(data_get_params or {}).get(environment_name), + server=server, **website, ) except httpx.HTTPStatusError as e: body_snippet = "" @@ -671,6 +710,9 @@ async def export_one(environment_name: str): return public, f"{public}: {e}" except ValueError as e: return str(e), str(e) + if result is None: + no_data_plane.add(environment_name) + return None if isinstance(result, tuple): try: @@ -744,7 +786,7 @@ def _register_uploaded_bundle(): except OSError: pass - results = await asyncio.gather(*(export_one(name) for name in services)) + results = await asyncio.gather(*(export_one(server) for server in servers)) environment_artifacts = [] skipped: dict[str, tuple[str, str]] = {} @@ -767,9 +809,18 @@ def _register_uploaded_bundle(): f"{len(skipped)}/{len(services)} services were not exported " f"(missing={sorted(skipped)}): {public_summary}" ) - if empty := [name for name, result in zip(services, results) if result is None]: - if len(empty) == len(services): - raise RuntimeError(f"{', '.join(map(repr, empty))} returned no files to snapshot") + without_data_plane = [name for name in services if name in no_data_plane] + if without_data_plane: + logger.warning( + f"snapshot_env: {without_data_plane} of {env.id} serve no v1 data plane; " + "their state is not in the snapshot" + ) + empty = [name for name, result in zip(services, results) if result is None and name not in no_data_plane] + if not environment_artifacts: + reasons = [f"{without_data_plane} serve no v1 data plane"] if without_data_plane else [] + reasons += [f"{', '.join(map(repr, empty))} returned no files to snapshot"] if empty else [] + raise RuntimeError(f"Env '{env.id}' v{env.version} exported no state: {'; '.join(reasons)}") + if empty: logger.info(f"snapshot_env: {empty} returned no files to keep, so {snapshot_id} leaves them out") universe = await asyncio.to_thread( @@ -849,25 +900,6 @@ def registered_upload(object_url: str, file_id: str, filename: str, content_type id=file_id, description=description, filename=filename, content_type=content_type, object_url=object_url)) -def _card_environments(deployed) -> list[str]: - """The environments a deployment without a gateway serves, as the card it recorded names them: the env itself when - it has no children; else each child its name reaches that exports, and the env too, first, when it exports beside - them, as a computer fronting its apps does.""" - card = deployed.environment_card or {} - children = card.get("children_environments") - if not isinstance(children, list): - names = [card.get("name")] - else: - names = [child.get("name") for child in children if deployed.get_child_env_card(child.get("name")) is child - and ((operations := protocol_v1.operations(child)) is None - or METHOD_GET in operations)] - if METHOD_GET in (protocol_v1.operations(card) or []): - names.insert(0, card.get("name")) - if not all(names): - raise RuntimeError(f"Deployed env '{deployed.env_id}' recorded a card naming no environment to snapshot") - return names - - def _changes_route(deployed, override_url: Optional[str]) -> tuple[str, str]: """The verb and URL of the env's changes: an override keeps `{override}/changes`, else the stored card names them.""" if override_url: diff --git a/src/agent_env/task_step/task_steps/sync_env_clock.py b/src/agent_env/task_step/task_steps/sync_env_clock.py index 76ff196e8..8ae463f56 100644 --- a/src/agent_env/task_step/task_steps/sync_env_clock.py +++ b/src/agent_env/task_step/task_steps/sync_env_clock.py @@ -1,8 +1,9 @@ """Arm a deployed env gateway's virtual clock, then sync its backing servers to it. Arms the clock (set_time) and reads the gateway's server-reachable env_get_time_url (state), both as the -stored env card declares clock/v1, then invokes sync_time on each backing server whose child env card -advertises it. Non-advertisers are skipped when tolerate_missing_sync_time (default True). Place right +stored env card declares clock/v1, then invokes sync_time on each backing server, MCP server or website, the +env card lists (``legacy_protocol.backing_servers``) whose own card advertises it, at the route that card gives. +Non-advertisers are skipped when tolerate_missing_sync_time (default True), and websites always. Place right before prompt_agent so the virtual timeline starts at agent-start. """ @@ -10,7 +11,7 @@ import logging from datetime import datetime, timezone as _utc_tz -from typing import ClassVar, Optional +from typing import TYPE_CHECKING, ClassVar, Optional from zoneinfo import ZoneInfo, ZoneInfoNotFoundError import httpx @@ -22,6 +23,9 @@ from agent_env.entity.refs import EntityRef from agent_env.task_step.task_step import TaskStep, TaskStepDependency +if TYPE_CHECKING: + from agent_env.env.legacy_protocol import BackingServer + logger = logging.getLogger(__name__) @@ -107,14 +111,13 @@ def from_dict(cls, data: dict) -> SyncEnvClockTaskStep: timezone=data.get("timezone"), ) - def _service_names(self, deployed) -> list[str]: - """Backing MCP server names for the env (mirrors apply_server_config's enumeration).""" - from agent_env.env.env import Env - env = Env.get(deployed.env_id, deployed.env_version) - names = [e.environment_name for e in (getattr(env, "mcp_server_envs", None) or [])] - if not names and getattr(env, "environment_name", None): - names = [env.environment_name] - return names + async def _backing_servers(self, deployed) -> list[BackingServer]: + """The env's backing MCP servers and websites, as its card lists them (``legacy_protocol.backing_servers``): + a website sharing its name with an MCP server is a server of its own, at its own route.""" + from agent_env.env import legacy_protocol + from agent_env.env.env import Env, gateway_url_of + return await legacy_protocol.backing_servers(deployed, gateway_url_of(deployed), Env.get(deployed.env_id, deployed.env_version), + timeout=self.timeout_seconds) async def execute(self, context: TaskStepContext) -> TaskStepContext: from agent_env.env.env import gateway_url_of @@ -132,9 +135,12 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: env_get_time_url = state.get("env_get_time_url") synced, skipped = [], [] - for service in self._service_names(deployed): - outcome = await self._sync_one(deployed, gateway_url_of(deployed), service, env_get_time_url) - (synced if outcome.get("synced") else skipped).append({"service": service, "environment": service, **outcome}) + for server in await self._backing_servers(deployed): + service = server.environment_name + outcome = await self._sync_one(deployed, gateway_url_of(deployed), service, env_get_time_url, mcp=server.mcp, + tolerate=self.tolerate_missing_sync_time or not server.mcp, server=server) + entry = {"service": service, "environment": service, **({} if server.mcp else {"website": True}), **outcome} + (synced if outcome.get("synced") else skipped).append(entry) configuration = { "step_id": self.id, "env_id": self.env_id, @@ -151,17 +157,27 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: f"({[(s['service'], s.get('reason')) for s in skipped]}) — every tool still answers wall time") return context - async def _sync_one(self, deployed, gateway_url: str, service: str, env_get_time_url: Optional[str]) -> dict: - """Invoke sync_time on one server if its card advertises clock/v1; skip (or raise) per the flag.""" + async def _sync_one(self, deployed, gateway_url: str, service: str, env_get_time_url: Optional[str], mcp: bool = True, + tolerate: Optional[bool] = None, server: Optional[BackingServer] = None) -> dict: + """Invoke sync_time on one server, an MCP server or with ``mcp=False`` a website, if its card advertises + clock/v1; skip a server with no card or no clock/v1 when ``tolerate`` (tolerate_missing_sync_time by default), + else raise. ``execute`` tolerates every website, so one that doesn't follow the clock never fails the step; one + that follows it but can't take the task's timezone does, as an MCP server does. ``server`` is the server as the + env's card lists it (``legacy_protocol.server_card``); without it, the card is ``service``'s on ``deployed``'s + stored card, or read live at its gateway path.""" + tolerate = self.tolerate_missing_sync_time if tolerate is None else tolerate from agent_env.env import legacy_protocol - base_url, card = await legacy_protocol.child_env_card(deployed, gateway_url, service, timeout=self.timeout_seconds) + if server is not None: + base_url, card = await legacy_protocol.server_card(server, gateway_url, timeout=self.timeout_seconds) + else: + base_url, card = await legacy_protocol.child_env_card(deployed, gateway_url, service, timeout=self.timeout_seconds, mcp=mcp) if card is None: - if self.tolerate_missing_sync_time: + if tolerate: return {"synced": False, "reason": "no_env_card"} raise RuntimeError(f"{service} has no env card: it serves none, its card isn't named {service!r}, or the deploy couldn't read it") if protocol_v1.find_extension(card, EXT_CLOCK_URI) is None: - if self.tolerate_missing_sync_time: + if tolerate: logger.info(f"sync_env_clock: {service} does not advertise {EXT_CLOCK_URI}; skipping") return {"synced": False, "reason": "not_advertised"} raise RuntimeError(f"{service} does not advertise {EXT_CLOCK_URI}") @@ -176,5 +192,6 @@ async def _sync_one(self, deployed, gateway_url: str, service: str, env_get_time params["timezone"] = self.timezone result = await protocol_v1.invoke_extension(base_url, card, EXT_CLOCK_URI, params=params, - timeout=self.timeout_seconds, headers=deployed.control_headers) + timeout=self.timeout_seconds, + headers=deployed.control_headers) return {"synced": True, "result": result} diff --git a/src/agent_env/task_step/task_steps/traffic_capture.py b/src/agent_env/task_step/task_steps/traffic_capture.py new file mode 100644 index 000000000..f6dd48cd3 --- /dev/null +++ b/src/agent_env/task_step/task_steps/traffic_capture.py @@ -0,0 +1,235 @@ +"""Collect a website env's traffic captures. A website or multi env deployed with ``capture_traffic`` records every +packet its websites' containers send and receive, from just after each starts, as rotating pcap files, which its gateway's card serves as traffic/v1 +behind the deploy's control token. ``collect_traffic_capture`` stores each as a FileArtifact and registers them as one +FileArtifactUniverse, recorded in ``context.metadata[TRAFFIC_KEY][env_id]``. Teardown collects any capturing env a run +didn't, a retry's rolled-back ones included, before the env closes (``collect_uncollected_captures``), so a capture +isn't lost with its VM, and saves what it recorded to the run's stored instance.""" +from __future__ import annotations + +import asyncio +import copy +import functools +import logging +import os +import tempfile +from collections.abc import Sequence +from typing import ClassVar, Optional + +from agentenv_protocol.traffic import ( + PCAP_MEDIA_TYPE, + TRAFFIC_EXTENSION_URI, + TRAFFIC_LIST_OP, + TrafficCapture, + fetch_traffic_capture, + list_traffic_captures, + serves_traffic, +) + +from agent_env.artifact import FileArtifact +from agent_env.artifact.artifacts.file_artifact_universe import FileArtifactUniverse +from agent_env.artifact.store import get_artifact_store +from agent_env.config import get_config +from agent_env.entity.refs import EntityRef +from agent_env.env.env import DeployedEnv, EnvCapabilityUnsupported +from agent_env.store.ids import derive_id, is_local_id, validate_local_id +from agent_env.task_step.context import TaskStepContext +from agent_env.task_step.task_step import TaskStep, TaskStepDependency +from agent_env.task_step.thread_work import finish_on_thread + +logger = logging.getLogger(__name__) + +#: ``context.metadata[TRAFFIC_KEY][env_id]``: each env whose captures a run collected, or tried to. +TRAFFIC_KEY = "traffic_captures" +_ARTIFACT_TYPE = "traffic_captures" +_SAVING: set[asyncio.Future] = set() +_ERROR_CHARS = 500 + + +class CollectTrafficCaptureTaskStep(TaskStep): + """Store the traffic captures of the deployed env ``env_id`` as pcap FileArtifacts. Best effort by default: a + failure is recorded and logged, and fails the task only with ``fail_task_on_error``.""" + + type: ClassVar[str] = "collect_traffic_capture" + entity_refs = (EntityRef.env("env_id"),) + + def __init__(self, id: str, version: Optional[int], env_id: str, + depends_on: Optional[list[TaskStepDependency]] = None, fail_task_on_error: bool = False): + super().__init__(id, version, depends_on=depends_on, fail_task_on_error=fail_task_on_error) + if not env_id: + raise ValueError("collect_traffic_capture needs the env_id of an env deployed with capture_traffic") + self.env_id = env_id + + def to_dict(self) -> dict: + return {**super().to_dict(), "env_id": self.env_id} + + @classmethod + def from_dict(cls, data: dict) -> CollectTrafficCaptureTaskStep: + base = cls._base_from_dict(data) + base["fail_task_on_error"] = data.get("fail_task_on_error", False) + return cls(**base, env_id=data["env_id"]) + + async def execute(self, context: TaskStepContext) -> TaskStepContext: + deployed = next((d for d in context.deployed_envs if d.env_id == self.env_id), None) + if deployed is None: + error = RuntimeError(f"env {self.env_id!r} isn't deployed in this run") + context.metadata.setdefault(TRAFFIC_KEY, {})[self.env_id] = _failed(error) + logger.warning("collect_traffic_capture: %s", error) + if self.fail_task_on_error: + raise error + return context + entry = await collect_captures(context, deployed) + if self.fail_task_on_error and entry["state"] != "collected": + raise RuntimeError(f"env {self.env_id!r}'s traffic captures weren't collected: {entry.get('error')}") + return context + + +def captures_traffic(deployed: DeployedEnv) -> bool: + """Whether the stored card of ``deployed`` serves traffic/v1, as a deploy with ``capture_traffic`` does; no + network.""" + return isinstance(deployed, DeployedEnv) and serves_traffic(deployed.environment_card) + + +async def collect_uncollected_captures(context: TaskStepContext, envs: Sequence[DeployedEnv], *, + rolled_back: Sequence[tuple[str, DeployedEnv]] = ()) -> None: + """Collect the captures of each of ``envs`` that serves traffic/v1 and whose captures the run hasn't collected, a + collection that failed tried again, as teardown does before the envs close, and of each ``rolled_back`` deployment, + a retry's, under its own key (``rolled_back_deployments``); ``collect_captures`` records what happened to each, and + the entries are saved to the run's stored instance, since teardown runs after its steps saved theirs, even when + teardown's time runs out partway, so what was collected by then stays linked to the run.""" + entries = context.metadata.get(TRAFFIC_KEY) or {} + tried = False + try: + for key, deployed in [*((deployed.env_id, deployed) for deployed in envs), *rolled_back]: + if captures_traffic(deployed) and (entries.get(key) or {}).get("state") != "collected": + tried = True + await collect_captures(context, deployed, key=key) + except BaseException: + if tried: + _save_in_background(context) + raise + if tried: + await _save(context) + + +async def _save(context: TaskStepContext) -> None: + """Save the run's ``traffic_captures`` entries to its stored instance on a worker thread, which finishes the + write even when teardown's time runs out while it waits.""" + if context.instance_id and TRAFFIC_KEY in context.metadata: + from agent_env.task.store import record_context_metadata + + await finish_on_thread(functools.partial(record_context_metadata, context.instance_id, TRAFFIC_KEY, + copy.deepcopy(context.metadata[TRAFFIC_KEY])), + f"Saving the traffic captures of task instance {context.instance_id}") + + +def _save_in_background(context: TaskStepContext) -> None: + """Start ``_save`` without waiting for it, as a collection teardown's time ran out on does, so the entries + collected by then are saved while teardown goes on to terminate the sandboxes.""" + saving = asyncio.ensure_future(_save(context)) + _SAVING.add(saving) + saving.add_done_callback(_SAVING.discard) + + +def rolled_back_deployments(context: TaskStepContext) -> list[tuple[str, DeployedEnv]]: + """Each capturing env a retry rolled back, with the key its captures are recorded under: + ``-rolled-back-`` for the ``n``-th of the run's ``retry_resets``.""" + found = [] + for n, reset in enumerate(context.metadata.get("retry_resets") or [], start=1): + for record in (reset.get("rolled_back_sandboxes") or {}).get("capturing_envs") or []: + deployed = DeployedEnv.from_dict(record) + found.append((f"{deployed.env_id}-rolled-back-{n}", deployed)) + return found + + +async def collect_captures(context: TaskStepContext, deployed: DeployedEnv, *, key: Optional[str] = None) -> dict: + """Store each capture ``deployed`` serves as a pcap FileArtifact, register them as one FileArtifactUniverse, and + record it in ``context.metadata[TRAFFIC_KEY][key]``, by default its env_id. Each file is named for its container + and its place among that container's files, oldest first, so they read in the order their packets were captured. + Never raises: the entry, which it also returns, says what happened. Each file stored is registered in the universe + as it lands, every store write on a worker thread, so a collection that stops partway, failed or cancelled, keeps + the files it stored by then recorded as ``partial``, which teardown collects again.""" + key = key or deployed.env_id + entries = context.metadata.setdefault(TRAFFIC_KEY, {}) + try: + if not captures_traffic(deployed): + raise EnvCapabilityUnsupported(TRAFFIC_EXTENSION_URI, TRAFFIC_LIST_OP, deployed.env_id, deployed.instance_id) + listing = await list_traffic_captures(deployed.environment_url, deployed.environment_card, + headers=deployed.control_headers) + entries[key] = await _store(context, deployed, listing.captures, key) + except Exception as e: + logger.warning("collect_traffic_capture %s: %s", key, e) + partial = entries.get(key) or {} + entries[key] = {**partial, **_failed(e), "state": "partial"} if partial.get("state") == "partial" else _failed(e) + return entries[key] + + +async def _store(context: TaskStepContext, deployed: DeployedEnv, captures: list[TrafficCapture], key: str) -> dict: + universe_id = derive_id(context.instance_id or "run", f"traffic-{key}") + if is_local_id(universe_id): + validate_local_id(universe_id) + get_config().check_local_run_write(universe_id) + store = get_artifact_store() + prefix = store.attempt_prefix(_ARTIFACT_TYPE, universe_id) + files, total, seen, artifact = {}, 0, {}, None + try: + for capture in captures: + container = capture.container + seen[container] = seen.get(container, -1) + 1 + filename = f"{container}-{seen[container]:02d}.pcap" + handle, path = tempfile.mkstemp(suffix=".pcap") + os.close(handle) + try: + total += await fetch_traffic_capture(deployed.environment_url, deployed.environment_card, capture.name, + path, headers=deployed.control_headers) + except BaseException: + _remove(path) + raise + url = await finish_on_thread(functools.partial(_upload, universe_id, prefix + filename, path), + f"Uploading traffic capture {filename} of env {deployed.env_id!r}", + if_never_run=functools.partial(_remove, path)) + document = FileArtifact( + id=derive_id(universe_id, filename), filename=filename, content_type=PCAP_MEDIA_TYPE, object_url=url, + description=f"Traffic capture of {container} in env {deployed.env_id!r}, task instance {context.instance_id}") + files[filename] = await finish_on_thread(functools.partial(store.put_document, document), + f"Storing traffic capture {filename} of env {deployed.env_id!r}") + artifact = await finish_on_thread(functools.partial(_register, universe_id, dict(files)), + f"Registering the traffic captures of env {deployed.env_id!r}") + context.metadata[TRAFFIC_KEY][key] = _entry("partial", artifact, files, total, seen) + except BaseException as e: + if artifact is not None: + context.metadata[TRAFFIC_KEY][key] = {**_entry("partial", artifact, files, total, seen), + "error": _failed(e)["error"]} + raise + logger.info("collect_traffic_capture %s: %d capture files, %d bytes", deployed.env_id, len(files), total) + return _entry("collected", artifact, files, total, seen) + + +def _register(universe_id: str, files: dict) -> dict: + """Register ``files`` as the next version of the FileArtifactUniverse ``universe_id``.""" + universe = FileArtifactUniverse.put(id=universe_id, file_artifacts=files) + return {"id": universe.id, "version": universe.version} + + +def _entry(state: str, artifact: Optional[dict], files: dict, total: int, seen: dict) -> dict: + return {"state": state, "artifact": artifact, "files": sorted(files), "bytes": total, "containers": sorted(seen)} + + +def _upload(universe_id: str, object_url: str, path: str) -> str: + """Upload the capture at ``path`` to ``object_url``, write-once, and remove it: the upload owns the file, so a + cancelled collection never removes it from under the upload.""" + try: + return get_config().get_object_store_to_write(object_url, universe_id).put_file_at(object_url, path, + PCAP_MEDIA_TYPE) + finally: + _remove(path) + + +def _remove(path: str) -> None: + try: + os.unlink(path) + except FileNotFoundError: + pass + + +def _failed(error: BaseException) -> dict: + return {"state": "failed", "error": f"{type(error).__name__}: {error}"[:_ERROR_CHARS]} diff --git a/src/agent_env/task_step/task_steps/verifiers/__init__.py b/src/agent_env/task_step/task_steps/verifiers/__init__.py index e3e284145..781c29c39 100644 --- a/src/agent_env/task_step/task_steps/verifiers/__init__.py +++ b/src/agent_env/task_step/task_steps/verifiers/__init__.py @@ -10,7 +10,6 @@ from agent_env.task_step.task_steps.verifiers.agent_prompt_response_verifier import AgentPromptResponseVerifierTaskStep from agent_env.task_step.task_steps.verifiers.run_container_unit_tests_verifier import RunContainerUnitTestsVerifierTaskStep from agent_env.task_step.task_steps.verifiers.verify_sandbox import VerifySandboxTaskStep -from agent_env.task_step.task_steps.verifiers.website_state_verifier import WebsiteStateVerifierTaskStep from agent_env.task_step.task_steps.verifiers.aggregate_verifiers import AggregateVerifiersTaskStep __all__ = [ @@ -27,6 +26,5 @@ "ScoreAggregator", "Turn", "VerifySandboxTaskStep", - "WebsiteStateVerifierTaskStep", "aggregate_score", ] diff --git a/src/agent_env/task_step/task_steps/verifiers/website_state_verifier.py b/src/agent_env/task_step/task_steps/verifiers/website_state_verifier.py deleted file mode 100644 index 2546c9141..000000000 --- a/src/agent_env/task_step/task_steps/verifiers/website_state_verifier.py +++ /dev/null @@ -1,217 +0,0 @@ -"""Verify the state of deployed websites, a website env or a multi env's: each check reads one value from one session -of one site, as the site's /go?sid= reports it ({initial_state, current_state, state_diff}, the verifier -endpoint CUA-Gym sites serve), reached through the gateway's /svc//, and compares it with an expected -value.""" - -from __future__ import annotations - -import logging -import re -import uuid - -import httpx -from dataclasses import asdict, dataclass -from typing import Any, ClassVar, Optional - -from agent_env.entity.refs import EntityRef -from agent_env.env import legacy_protocol -from agent_env.env.env import Env, gateway_url_of -from agent_env.store.base import NotFoundError -from agent_env.task_step.context import TaskStepContext -from agent_env.task_step.task_step import TaskStep, TaskStepDependency, attach_retry_config -from agent_env.task_step.task_steps.verifiers.scoring import ScoreAggregator, aggregate_score - -logger = logging.getLogger(__name__) - -SOURCES = ("current_state", "initial_state", "state_diff") -OPS = ("eq", "ne", "ge", "gt", "le", "lt", "in", "contains") -REDUCES = ("len", "keys", "bool") -DEFAULT_SID = "default" -GO_TIMEOUT_S = 30 - - -@dataclass(frozen=True) -class StateCheck: - """One check: ``op`` of the value at ``path`` (dot-separated, a key's ``.`` and ``\\`` escaped with ``\\``, a list - index by number; empty for the whole source) in session ``sid`` of ``site``, a website env's environment_name, - reduced by ``reduce`` first when given, and ``value``. A path that isn't there reads as None.""" - - site: str - path: str - op: str - value: Any = None - sid: str = DEFAULT_SID - source: str = "current_state" - reduce: Optional[str] = None - weight: float = 1.0 - - def problems(self) -> list[str]: - named = f"check of {self.site!r} {self.path!r}" - return ([f"{named}: op {self.op!r} is none of {', '.join(OPS)}"] if self.op not in OPS else []) + ( - [f"{named}: source {self.source!r} is none of {', '.join(SOURCES)}"] if self.source not in SOURCES else []) + ( - [f"{named}: reduce {self.reduce!r} is none of {', '.join(REDUCES)}"] - if self.reduce is not None and self.reduce not in REDUCES else []) - - def result(self, sessions: dict) -> dict: - """The check's result row against ``sessions``, each site's ``/go`` answer by its sid.""" - session = sessions.get(self.site, {}).get(self.sid) - actual = None if session is None else _read(session.get(self.source), self.path, self.source == "state_diff") - if self.reduce is not None: - actual = _reduce(actual, self.reduce) - try: - passed = _compare(actual, self.op, self.value) - except TypeError: - passed = False - return {"criterion": f"{self.site}[{self.sid}].{self.source}{'.' + self.path if self.path else ''}" - f"{' |' + self.reduce if self.reduce else ''} {self.op} {self.value!r}", - "result": passed, "actual": actual, "weight": self.weight, - **({} if session is not None else {"detail": f"{self.site!r} has no session {self.sid!r}"})} - - -def _read(source: Any, path: str, whole_keys: bool) -> Any: - if not path: - return source - if whole_keys: - return source.get(path) if isinstance(source, dict) else None - value = source - for part in _parts(path): - if isinstance(value, dict): - value = value.get(part) - elif isinstance(value, list) and re.fullmatch(r"-?\d+", part) and -len(value) <= int(part) < len(value): - value = value[int(part)] - else: - return None - return value - - -def _parts(path: str) -> list[str]: - """``path``'s keys, split at each ``.`` that no ``\\`` escapes, as the runtime escapes a key's ``.`` and ``\\``.""" - parts, part, chars = [], [], iter(path) - for char in chars: - if char == "\\": - part.append(next(chars, "")) - elif char == ".": - parts.append("".join(part)) - part = [] - else: - part.append(char) - return [*parts, "".join(part)] - - -def _reduce(value: Any, how: str) -> Any: - if how == "bool": - return bool(value) - if how == "keys": - return sorted(value) if isinstance(value, dict) else [] - return len(value) if isinstance(value, (dict, list, str)) else 0 - - -def _compare(actual: Any, op: str, expected: Any) -> bool: - if op == "eq": - return actual == expected - if op == "ne": - return actual != expected - if op == "in": - return actual in expected - if op == "contains": - return actual is not None and expected in actual - if actual is None or expected is None: - return False - return {"ge": actual >= expected, "gt": actual > expected, "le": actual <= expected, "lt": actual < expected}[op] - - -class WebsiteStateVerifierTaskStep(TaskStep): - """Runs ``checks`` against the state the websites of the deployed env ``env_id``, a website env or a multi env, - report at ``/go?sid=``, one result row each, and scores them with ``score_aggregator``. The sites checks name are - checked before a run against ``env_version``, the version the task deploys, when it's given.""" - - type: ClassVar[str] = "website_state_verifier" - entity_refs = (EntityRef.env("env_id", version_field="env_version"),) - - def __init__(self, id: str, version: Optional[int], env_id: str, checks: list[StateCheck | dict], - score_aggregator: Optional[ScoreAggregator | str] = None, verifier_id: Optional[str] = None, - depends_on: Optional[list[TaskStepDependency]] = None, fail_task_on_error: bool = True, - env_version: Optional[int] = None): - super().__init__(id, version, depends_on=depends_on, fail_task_on_error=fail_task_on_error) - self.env_id = env_id - self.env_version = env_version - self.checks = [check if isinstance(check, StateCheck) else StateCheck(**check) for check in checks] - self.score_aggregator = ScoreAggregator(score_aggregator) if isinstance(score_aggregator, str) else ( - score_aggregator or ScoreAggregator.ALL_PASS) - self.verifier_id = verifier_id or uuid.uuid4().hex - if problems := [problem for check in self.checks for problem in check.problems()]: - raise ValueError(f"website_state_verifier {id!r}: " + "; ".join(problems)) - - def to_dict(self) -> dict: - return {**super().to_dict(), "env_id": self.env_id, "env_version": self.env_version, - "checks": [asdict(check) for check in self.checks], "score_aggregator": self.score_aggregator.value, - "verifier_id": self.verifier_id} - - @classmethod - def from_dict(cls, data: dict) -> WebsiteStateVerifierTaskStep: - return cls(**cls._base_from_dict(data), env_id=data["env_id"], env_version=data.get("env_version"), - checks=data["checks"], score_aggregator=data.get("score_aggregator"), - verifier_id=data.get("verifier_id")) - - def preflight(self) -> list[str]: - """The env ``env_id`` when it holds no website, and the sites checks name that it doesn't serve, read from - ``env_version``; with no ``env_version`` the sites are left to the run, which reads the version it deploys.""" - try: - env = Env.get(self.env_id, self.env_version) - except NotFoundError: - return [] - if not (sites := _sites(env)): - return [f"website_state_verifier {self.id!r}: {self.env_id} is a {env.type} env with no website; name a " - "website env or a multi env of websites"] - if self.env_version is None: - return [] - if unknown := sorted({check.site for check in self.checks} - sites): - return [f"website_state_verifier {self.id!r}: {', '.join(map(repr, unknown))} " - f"{'is not a site' if len(unknown) == 1 else 'are not sites'} of {self.env_id} " - f"v{self.env_version}, whose sites are {sorted(sites)}"] - return [] - - @classmethod - def put(cls, **kwargs: Any) -> WebsiteStateVerifierTaskStep: - """Store the step once its preflight passes. Refused, naming every problem, before anything is stored.""" - from agent_env.task_step.store import get_task_step_store - - kwargs.setdefault("version", None) - retry_config = kwargs.pop("retry_config", None) - step = cls(**kwargs) - if problems := step.preflight(): - raise ValueError("; ".join(problems)) - if retry_config is not None: - attach_retry_config(step, {"retry_config": retry_config}) - return get_task_step_store().put_document(step) - - async def execute(self, context: TaskStepContext) -> TaskStepContext: - deployed = next((d for d in context.deployed_envs if d.env_id == self.env_id), None) - if deployed is None: - raise RuntimeError(f"Env '{self.env_id}' not found in context.deployed_envs") - sites = _sites(Env.get(deployed.env_id, deployed.env_version)) - gateway_url = gateway_url_of(deployed) - sessions: dict[str, dict] = {} - async with httpx.AsyncClient() as client: - for site, sid in sorted({(check.site, check.sid) for check in self.checks}): - if site not in sites: - raise RuntimeError(f"env '{self.env_id}' serves no website {site!r}; its sites are {sorted(sites)}") - base_url = legacy_protocol.environment_base_url(gateway_url, site, mcp=False) - response = await client.get(f"{base_url}/go", params={"sid": sid}, timeout=GO_TIMEOUT_S) - response.raise_for_status() - sessions.setdefault(site, {})[sid] = response.json() - results = [check.result(sessions) for check in self.checks] - score = aggregate_score(results, self.score_aggregator) - context.metadata.setdefault("verifications", {})[self.verifier_id] = {"results": results, "score": score} - logger.info(f"Verification '{self.verifier_id}' complete: {len(results)} checks, score={score}") - return context - - -def _sites(env: Env) -> set[str]: - """The environment_names of the website envs ``env`` is or holds.""" - from agent_env.env.envs.multi_env import MultiEnv - from agent_env.env.envs.website import WebsiteEnv - - if isinstance(env, WebsiteEnv): - return {env.environment_name} - return {child.environment_name for child in env.website_envs} if isinstance(env, MultiEnv) else set() diff --git a/tst/integration/env/gateway/gateway_test.py b/tst/integration/env/gateway/gateway_test.py index ab4057715..eefe02c96 100644 --- a/tst/integration/env/gateway/gateway_test.py +++ b/tst/integration/env/gateway/gateway_test.py @@ -780,7 +780,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) cli_list = await client.post( f"{result.gateway_url}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "cli"}, + headers=result.role_headers("cli"), timeout=30, ) cli_list.raise_for_status() @@ -797,7 +797,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) cli_call = await client.post( f"{result.gateway_url}/step", json={"action": "call_tool", "tool_name": "list_emails", "arguments": {"folder_name": "INBOX"}}, - headers={AGENT_ENV_ROLE_HEADER: "cli"}, + headers=result.role_headers("cli"), timeout=30, ) assert cli_call.status_code == 200, f"cli call failed: {cli_call.status_code} {cli_call.text}" @@ -814,7 +814,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) async with streamable_http_client( result.mcp_url, - http_client=httpx.AsyncClient(headers={AGENT_ENV_ROLE_HEADER: "cli"}), + http_client=httpx.AsyncClient(headers=result.role_headers("cli")), ) as (read_stream, write_stream, _): async with ClientSession(read_stream, write_stream) as session: await session.initialize() @@ -826,7 +826,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) assert "list_emails" not in [t.name for t in await default_session.list_tools()] hidden = await default_session.call_tool("list_emails", {"folder_name": "INBOX"}) assert hidden.isError and "is disabled for role 'default'" in hidden.text, hidden - async with ToolSession(mcp_url=result.mcp_url, role="cli") as cli_session: + async with ToolSession(mcp_url=result.mcp_url, headers=result.role_headers("cli")) as cli_session: assert "list_emails" in [t.name for t in await cli_session.list_tools()] assert "alex.chen@techcorp.com" in (await cli_session.call_tool("list_emails", {"folder_name": "INBOX"})).text @@ -869,7 +869,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) assert state_resp.json()["roles"]["*"] == {"disabled": ["list_emails"], "allowed": []} for role in (None, "cli", "marketing"): - headers = {AGENT_ENV_ROLE_HEADER: role} if role else {} + headers = result.role_headers(role) if role else {} lst = await client.post( f"{result.gateway_url}/step", json={"action": "list_tools"}, headers=headers, timeout=30, ) @@ -879,14 +879,14 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) blocked = await client.post( f"{result.gateway_url}/step", json={"action": "call_tool", "tool_name": "list_emails", "arguments": {"folder_name": "INBOX"}}, - headers={AGENT_ENV_ROLE_HEADER: "marketing"}, + headers=result.role_headers("marketing"), timeout=30, ) assert blocked.status_code == 403, f"expected 403 for never-seen role under global disable, got {blocked.status_code} {blocked.text}" async with streamable_http_client( result.mcp_url, - http_client=httpx.AsyncClient(headers={AGENT_ENV_ROLE_HEADER: "cli"}), + http_client=httpx.AsyncClient(headers=result.role_headers("cli")), ) as (read_stream, write_stream, _): async with ClientSession(read_stream, write_stream) as session: await session.initialize() @@ -926,7 +926,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) mkt_lst = await client.post( f"{result.gateway_url}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "marketing"}, + headers=result.role_headers("marketing"), timeout=30, ) mkt_lst.raise_for_status() @@ -944,7 +944,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) cli_lst = await client.post( f"{result.gateway_url}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "cli"}, + headers=result.role_headers("cli"), timeout=30, ) cli_lst.raise_for_status() @@ -955,7 +955,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) mkt_lst = await client.post( f"{result.gateway_url}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "marketing"}, + headers=result.role_headers("marketing"), timeout=30, ) mkt_lst.raise_for_status() @@ -970,7 +970,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) cli_lst_after_reset = await client.post( f"{result.gateway_url}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "cli"}, + headers=result.role_headers("cli"), timeout=30, ) cli_lst_after_reset.raise_for_status() @@ -1002,7 +1002,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) blocked = await client.post( f"{result.gateway_url}/step", json={"action": "call_tool", "tool_name": "list_emails", "arguments": {"folder_name": "INBOX"}}, - headers={AGENT_ENV_ROLE_HEADER: "step-role"}, + headers=result.role_headers("step-role"), timeout=30, ) assert blocked.status_code == 403, f"expected 403 after step-driven disable, got {blocked.status_code} {blocked.text}" @@ -1020,7 +1020,7 @@ async def assert_generated_data_via_step(client, gateway_url, tools, tool_names) unblocked = await client.post( f"{result.gateway_url}/step", json={"action": "call_tool", "tool_name": "list_emails", "arguments": {"folder_name": "INBOX"}}, - headers={AGENT_ENV_ROLE_HEADER: "step-role"}, + headers=result.role_headers("step-role"), timeout=30, ) assert unblocked.status_code == 200, f"expected 200 after step-driven enable, got {unblocked.status_code} {unblocked.text}" @@ -1816,7 +1816,7 @@ async def test_gateway_changes_name_the_role_behind_each_write(mcp_server_envs, seen = (await client.get(f"{gw}/changes", params={"limit": 10000}, timeout=30)).json()["changes"] after_id = seen[-1]["id"] if seen else 0 for role, subject in (("default", "first"), ("alice@example.com", "second")): - resp = await client.post(f"{gw}/step", headers={AGENT_ENV_ROLE_HEADER: role}, timeout=60, json={ + resp = await client.post(f"{gw}/step", headers=result.role_headers(role), timeout=60, json={ "action": "call_tool", "tool_name": "send_email", "arguments": {"recipients": ["dana@example.com"], "subject": subject, "content": "body"}}) assert resp.status_code == 200, resp.text @@ -1850,7 +1850,7 @@ async def test_gateway_changes_name_the_request_behind_a_write_through_the_rest_ async with httpx.AsyncClient() as client: seen = (await client.get(f"{gw}/changes", params={"limit": 10000}, timeout=30)).json()["changes"] after_id = seen[-1]["id"] if seen else 0 - async with httpx.AsyncClient(headers={AGENT_ENV_ROLE_HEADER: "carol@example.com"}, timeout=60) as as_carol: + async with httpx.AsyncClient(headers=result.role_headers("carol@example.com"), timeout=60) as as_carol: async with streamable_http_client(f"{gw}/svc/mcp-email/mcp", http_client=as_carol) as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() @@ -2170,7 +2170,7 @@ async def assert_tool_sees_items(session, tools_result, tool_names): await _verify_list_tools(mcp_url=result.mcp_url, tool_verifier_callback=assert_tool_sees_items) # The role a ToolSession sends reaches the child's tool as its caller, with the gateway's key for each session - async with (ToolSession(mcp_url=result.mcp_url, role="alice@example.com") as alice, + async with (ToolSession(mcp_url=result.mcp_url, headers=result.role_headers("alice@example.com")) as alice, ToolSession(mcp_url=result.mcp_url) as anonymous): seen = [json.loads((await session.call_tool("items_whoami")).text) for session in (alice, alice, anonymous)] assert [caller["role"] for caller in seen] == ["alice@example.com", "alice@example.com", None] @@ -2548,7 +2548,7 @@ async def _poll(client, gw, want_fired, timeout_s=120): hr = await client.post(f"{gw}/step", json={"action": "call_tool", "tool_name": "search_emails", "arguments": {"query": "alex.chen", "folder_name": "INBOX"}}, - headers={AGENT_ENV_ROLE_HEADER: "harness"}, timeout=60) + headers=result.role_headers("harness"), timeout=60) assert hr.status_code == 200, f"harness /step failed: {hr.text}" await asyncio.sleep(4) assert {t["id"]: t["status"] for t in (await _state(client, gw))["triggers"]}["decoy"] == "armed", ( @@ -3064,7 +3064,7 @@ async def test_gateway_triggers_nl_executor(sandbox_provider, mcp_server_envs, e # The one-tool constraint really held (determinism lever, verified live). lt = await client.post(f"{gw}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "executor"}, timeout=30) + headers=result.role_headers("executor"), timeout=30) exec_tools = [t["name"] for t in lt.json()["tools"]] assert exec_tools == ["send_email"], f"executor tool surface not locked to one tool: {exec_tools}" @@ -3105,7 +3105,7 @@ async def _state(): se = await client.post(f"{gw}/step", json={"action": "call_tool", "tool_name": "search_emails", "arguments": {"query": marker, "folder_name": "SENT"}}, - headers={AGENT_ENV_ROLE_HEADER: "harness"}, timeout=30) + headers=result.role_headers("harness"), timeout=30) assert marker in se.text, f"marker email not found in SENT: {se.text[:400]}" # Cascade-safety: nlact was detected exactly once (from the provoke), NOT from the @@ -3351,17 +3351,17 @@ async def _rate_check(client, gw, rate, sleep_s=5.0): await client.post(f"{gw}/tools/{TOOL_DISABLE_ACTION}", json={"role": "locked", "tools": ["*"]}, timeout=30) locked = await client.post(f"{gw}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "locked"}, timeout=30) + headers=result.role_headers("locked"), timeout=30) assert GET_TIME not in [t["name"] for t in locked.json()["tools"]], "wildcard deny did not hide get_time" denied = await client.post(f"{gw}/step", json={"action": "call_tool", "tool_name": GET_TIME, "arguments": {}}, - headers={AGENT_ENV_ROLE_HEADER: "locked"}, timeout=30) + headers=result.role_headers("locked"), timeout=30) assert denied.status_code == 403, denied.text assert GET_TIME in (await _get_time_surfaces(client, gw, result.mcp_url)).step, "default role lost it" await client.post(f"{gw}/tools/{TOOL_ENABLE_ACTION}", json={"role": "locked", "tools": [GET_TIME]}, timeout=30) relocked = await client.post(f"{gw}/step", json={"action": "list_tools"}, - headers={AGENT_ENV_ROLE_HEADER: "locked"}, timeout=30) + headers=result.role_headers("locked"), timeout=30) assert GET_TIME in [t["name"] for t in relocked.json()["tools"]], "re-enable did not restore it" finally: await env.close() diff --git a/tst/integration/task_step/test_next_env_event_hook_local.py b/tst/integration/task_step/test_next_env_event_hook_local.py index de63dc0d7..103127090 100644 --- a/tst/integration/task_step/test_next_env_event_hook_local.py +++ b/tst/integration/task_step/test_next_env_event_hook_local.py @@ -12,7 +12,6 @@ import agent_env from agent_env.a2a_agent import conversation_store from agent_env.artifact import FileArtifact -from agent_env.env.gateway import AGENT_ENV_ROLE_HEADER from agent_env.task_step.context import TaskStepContext from agent_env.task_step.task_steps.prompt_agent import PromptAgentTaskStep from agent_env.task_step.task_steps.register_env_triggers import RegisterEnvTriggersStep @@ -65,7 +64,7 @@ async def _run(label: str, triggers: list[dict], params: dict) -> tuple: state = (await client.get("/triggers/state")).raise_for_status().json() clock = (await client.get("/clock/state")).raise_for_status().json() stored = await client.post("/step", json={"action": "call_tool", "tool_name": "list_items", "arguments": {}}, - headers={AGENT_ENV_ROLE_HEADER: "harness"}) + headers=deployed.role_headers("harness")) finally: await env.close() items = json.loads(stored.json()["content"][0]["text"])["items"] diff --git a/tst/integration/task_step/test_register_env_triggers_local.py b/tst/integration/task_step/test_register_env_triggers_local.py index 0a695d011..7c6964cca 100644 --- a/tst/integration/task_step/test_register_env_triggers_local.py +++ b/tst/integration/task_step/test_register_env_triggers_local.py @@ -17,9 +17,9 @@ import pytest import pytest_asyncio +from agent_env.env.env import DeployedEnv from agent_env.env.envs.mcp_server import MCPServerEnv from agent_env.env.envs.multi_env import MultiEnv -from agent_env.env.gateway import AGENT_ENV_ROLE_HEADER from agent_env.env.gateway.constants import METHOD_RESET, RPC_PATH, TRIGGER_IN_FLIGHT_STATUSES from agent_env.task_step.context import DeployedAgent, TaskStepContext from agent_env.task_step.task_steps.register_env_triggers import RegisterEnvTriggersStep @@ -66,8 +66,8 @@ async def test_each_nl_action_reaches_the_executor_it_names(items_env): {"id": "on-add", "when": {"type": "action", "tool": "items_add_item", "repeat": True}, "actions": []}, ] try: - alice = _Executor("alice", "alice@example.com", deployed.gateway_url, host, started) - bob = _Executor("bob", "bob@example.com", deployed.gateway_url, host, started) + alice = _Executor("alice", "alice@example.com", deployed, host, started) + bob = _Executor("bob", "bob@example.com", deployed, host, started) context = TaskStepContext(deployed_envs=[deployed], deployed_agents=[alice.agent, bob.agent], metadata={}) await RegisterEnvTriggersStep(id="reg", version=None, env_id=env.id, triggers=triggers, watch_roles=["default"], executor_timeout_seconds=60).execute(context) @@ -76,7 +76,7 @@ async def test_each_nl_action_reaches_the_executor_it_names(items_env): assert provoke.status_code == 200, provoke.text state = await _quiet(client, {"to-alice": 1, "to-bob": 1, "on-add": 0}) stored = await client.post("/step", json={"action": "call_tool", "tool_name": "list_items", "arguments": {}}, - headers={AGENT_ENV_ROLE_HEADER: "harness"}) + headers=deployed.role_headers("harness")) finally: for executor in started: executor.stop() @@ -108,8 +108,8 @@ async def test_personas_react_to_each_other_until_one_stays_quiet(items_env): "actions": [{"type": "nl", "executor": "denise", "instruction": "DENISE-TO-BOB"}]}, ] try: - denise = _Executor("denise", "denise@example.com", deployed.gateway_url, host, started) - bob = _Executor("bob", "bob@example.com", deployed.gateway_url, host, started, replies=1) + denise = _Executor("denise", "denise@example.com", deployed, host, started) + bob = _Executor("bob", "bob@example.com", deployed, host, started, replies=1) context = TaskStepContext(deployed_envs=[deployed], deployed_agents=[denise.agent, bob.agent], metadata={}) await RegisterEnvTriggersStep(id="reg", version=None, env_id=env.id, triggers=triggers, watch_roles=["default"], executor_timeout_seconds=60).execute(context) @@ -118,7 +118,7 @@ async def test_personas_react_to_each_other_until_one_stays_quiet(items_env): assert provoke.status_code == 200, provoke.text state = await _quiet(client, {"denise-hears-solver": 1, "bob-hears-denise": 2, "denise-hears-bob": 1}) stored = await client.post("/step", json={"action": "call_tool", "tool_name": "list_items", "arguments": {}}, - headers={AGENT_ENV_ROLE_HEADER: "harness"}) + headers=deployed.role_headers("harness")) finally: for executor in started: executor.stop() @@ -140,7 +140,7 @@ async def test_a_reply_waits_its_delay_and_hours_and_its_writes_name_the_fire(it triggers = [{"id": "bob-hears-solver", "when": {"type": "action", "tool": "list_items"}, "actions": [{"type": "nl", "executor": "bob", "instruction": "BOB-ACTS", "delay": "PT2M"}]}] try: - bob = _Executor("bob", "bob@example.com", deployed.gateway_url, host, started) + bob = _Executor("bob", "bob@example.com", deployed, host, started) context = TaskStepContext(deployed_envs=[deployed], deployed_agents=[bob.agent], metadata={}) async with httpx.AsyncClient(base_url=deployed.gateway_url, timeout=60) as client: # The env is shared, so its trajectory holds the earlier tests' calls: read this test's, after them. @@ -185,7 +185,7 @@ async def test_a_deadline_fires_on_silence_and_a_reply_cancels_it(items_env): "where": {"item": {"regex": f"^REPLY-{n}"}}}}, "actions": [{"type": "nl", "executor": "bob", "instruction": f"ESCALATE-{n}"}]}] try: - bob = _Executor("bob", "bob@example.com", deployed.gateway_url, host, started) + bob = _Executor("bob", "bob@example.com", deployed, host, started) context = TaskStepContext(deployed_envs=[deployed], deployed_agents=[bob.agent], metadata={}) async with httpx.AsyncClient(base_url=deployed.gateway_url, timeout=60) as client: await _arm_clock(client, "2026-10-14T10:00:00Z", 60) @@ -239,9 +239,10 @@ class _Executor: """An A2A executor on this machine that acts on each instruction through the gateway, under its own role; with ``replies``, it stops acting after that many.""" - def __init__(self, name: str, role: str, gateway_url: str, host: str, started: list, + def __init__(self, name: str, role: str, deployed: DeployedEnv, host: str, started: list, replies: int | None = None) -> None: - self.role, self.gateway_url, self.replies = role, gateway_url, replies + self.role, self.gateway_url, self.replies = role, deployed.gateway_url, replies + self.headers = deployed.role_headers(role) self.instructions: list[str] = [] self.acted_as: list[str] = [] self._server = ThreadingHTTPServer((host, 0), functools.partial(_ExecutorHandler, self)) @@ -265,7 +266,7 @@ def stop(self) -> None: def _call(self, tool: str, arguments: dict) -> dict: response = httpx.post(f"{self.gateway_url}/step", json={"action": "call_tool", "tool_name": tool, "arguments": arguments}, - headers={AGENT_ENV_ROLE_HEADER: self.role}, timeout=60) + headers=self.headers, timeout=60) response.raise_for_status() return response.json() diff --git a/tst/unit/bundle/env_toml_test.py b/tst/unit/bundle/env_toml_test.py index 236be7b99..8e8591735 100644 --- a/tst/unit/bundle/env_toml_test.py +++ b/tst/unit/bundle/env_toml_test.py @@ -155,6 +155,45 @@ def test_a_website_with_only_a_server_dockerfile_builds_one_image(bundle_dir, bu assert [call["build"][0] for call in builds] == ["Dockerfile.backend"] +SPEC = '[site]\nname = "mail"\ndomains = ["mail.test", "*.mail.test"]\nhealth = "/healthz"\n' + + +def test_a_website_takes_its_name_health_and_domains_from_the_spec_its_env_toml_names(bundle_dir, builds): + _bundle(bundle_dir, {"envs/mail/Dockerfile.backend": DOCKERFILE, "envs/mail/website.toml": SPEC, + "envs/mail/env.toml": 'type = "website"\nspec = "website.toml"\n'}, "mail") + + _run(bundle_dir) + env = Env.get(f"{ROOT}/mail") + + assert (env.environment_name, env.health_path, env.domains) == ("mail", "/healthz", ("mail.test", "*.mail.test")) + assert WebsiteEnv.from_dict(env.to_dict()).domains == env.domains + + +def test_a_website_takes_an_inline_spec_table(bundle_dir, builds): + _bundle(bundle_dir, {"envs/mail/Dockerfile.backend": DOCKERFILE, + "envs/mail/env.toml": 'type = "website"\n[spec.site]\nname = "mail"\ndomains = ["mail.test"]\n'}, + "mail") + + _run(bundle_dir) + env = Env.get(f"{ROOT}/mail") + + assert (env.environment_name, env.health_path, env.domains) == ("mail", "/api/health", ("mail.test",)) + + +@pytest.mark.parametrize("toml, spec, problem", [ + ('health = "/other"\n', SPEC, "envs/mail: env.toml: health '/other' isn't the spec's '/healthz'"), + ('environment_name = "post"\n', SPEC, "envs/mail: env.toml: environment_name 'post' isn't the spec's 'mail'"), + ("", SPEC + '[source.live]\nallow_hosts = ["cdn.test"]\n[users]\nsign_in = "live"\n', + "envs/mail: env.toml: spec: a live source isn't served by a server image, so a website env can't deploy it"), + ("", '[site]\ndomains = ["Mail.test"]\n', "site.domains: ['Mail.test'] aren't host names in lower case"), +], ids=["health", "name", "live", "invalid"]) +def test_a_website_whose_spec_disagrees_or_cant_deploy_is_refused(bundle_dir, toml, spec, problem): + _bundle(bundle_dir, {"envs/mail/Dockerfile.backend": DOCKERFILE, "envs/mail/website.toml": spec, + "envs/mail/env.toml": f'type = "website"\nspec = "website.toml"\n{toml}'}, "mail") + + assert any(problem in p for p in _problems(bundle_dir)), _problems(bundle_dir) + + def test_a_multi_names_bundle_and_store_envs_and_is_rewritten_when_a_child_is(bundle_dir, builds): with namespace_routing(): MCPServerEnv.put(id="inbox", docker_image_artifact=_store_image("inbox-image"), environment_name="inbox") diff --git a/tst/unit/cli/website_dev_cli_test.py b/tst/unit/cli/website_dev_cli_test.py new file mode 100644 index 000000000..e62a338f0 --- /dev/null +++ b/tst/unit/cli/website_dev_cli_test.py @@ -0,0 +1,311 @@ +"""agent-env env website dev's pieces, each without a long-running process: the site class it serves, the boot every +answer of a server carries, the local clock standing in for the gateway's, when web/ is rebuilt, priming a server +(clock/v1's sync_time and the seed through the data plane) once per boot with one conformance check, and the command +wiring uvicorn's reload, the clock and the supervisor together.""" + +import datetime +import json +import logging +import os +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import httpx +import pytest +from click.testing import CliRunner +from starlette.testclient import TestClient + +from agent_env.cli.env import website_dev +from agent_env.cli.env.website import website +from agent_env.cli.env.website_scaffold import scaffold +from agentenv_protocol.types import WELL_KNOWN_PATH +from agentenv_protocol.website import UNIVERSE_NOW_HEADER, conformance, website as declare +from agentenv_protocol.website.clock import parse_time + +TWO_SITES = ''' +from agentenv_protocol.website import WebsiteEnvironment, website + + +class Base(WebsiteEnvironment): + pass + + +@website(domains=["a.test"]) +class Site(Base): + pass +''' + + +@pytest.fixture +def site_dir(tmp_path: Path) -> Path: + scaffold(tmp_path / "shop", "shop") + page = tmp_path / "shop" / "web" / "dist" + page.mkdir() + (page / "index.html").write_text("shop") + return tmp_path / "shop" + + +@pytest.fixture +def clock(socket_enabled): + started = website_dev.DevClock(datetime.datetime(2030, 1, 1, 9, tzinfo=datetime.timezone.utc)).start() + yield started + started.close() + + +def _app(site_dir: Path, monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv(website_dev.SITE_ENV, str(site_dir / "site.py")) + return website_dev.create_app() + + +def test_the_site_class_is_the_one_its_file_defines_or_the_one_declared_with_website(tmp_path, site_dir): + assert website_dev.site_class(site_dir / "site.py").__name__ == "ShopSite" + (tmp_path / "two.py").write_text(TWO_SITES) + assert website_dev.site_class(tmp_path / "two.py").__name__ == "Site" + (tmp_path / "none.py").write_text("x = 1\n") + with pytest.raises(website_dev.DevError, match="defines 0 WebsiteEnvironment classes"): + website_dev.site_class(tmp_path / "none.py") + + +def test_a_site_imports_the_files_beside_it(tmp_path, monkeypatch): + monkeypatch.setattr(sys, "path", list(sys.path)) + (tmp_path / "shop").mkdir() + (tmp_path / "shop" / "shop_models_beside.py").write_text("DOMAIN = 'shop.test'\n") + (tmp_path / "shop" / "site.py").write_text( + "from agentenv_protocol.website import WebsiteEnvironment, website\n" + "from shop_models_beside import DOMAIN\n\n\n@website(domains=[DOMAIN])\nclass Shop(WebsiteEnvironment):\n pass\n") + monkeypatch.chdir(tmp_path) + assert website_dev.site_class(Path("shop") / "site.py").__name__ == "Shop" + + +def test_every_answer_carries_the_servers_boot_and_the_supervisors_polls_leave_the_access_log(site_dir, monkeypatch): + first, second = TestClient(_app(site_dir, monkeypatch)), TestClient(_app(site_dir, monkeypatch)) + boot = first.get("/healthz").headers[website_dev.BOOT_HEADER] + assert first.get("/api/items").headers[website_dev.BOOT_HEADER] == boot + assert second.get("/healthz").headers[website_dev.BOOT_HEADER] != boot + + def line(path): + return logging.LogRecord("uvicorn.access", logging.INFO, "", 0, '%s - "%s %s HTTP/%s" %d', + ("127.0.0.1:1", "GET", path, "1.1", 200), None) + assert not website_dev._unpolled(line(f"/.well-known/agent-env.json?{website_dev.POLL_QUERY}")) + assert website_dev._unpolled(line("/api/items")) + + +def test_the_local_clock_answers_as_the_gateways_and_is_set_and_advanced(clock): + base = clock.url.rsplit("/clock/", 1)[0] + assert parse_time(httpx.get(clock.url).json()["virtual_time"]) - parse_time("2030-01-01T09:00:00Z") < datetime.timedelta(seconds=5) + state = httpx.put(base + "/clock/set-time", json={"virtual_time": "2031-02-03T04:05:06Z", + "virtual_seconds_per_real_second": 0}).json() + assert (state["virtual_time"], state["virtual_seconds_per_real_second"]) == ("2031-02-03T04:05:06.000Z", 0.0) + assert httpx.put(base + "/clock/advance", json={"seconds": 3600}).json()["virtual_time"] == "2031-02-03T05:05:06.000Z" + assert httpx.put(base + "/clock/advance", json={"virtual_time": "2031-02-04T00:00:00Z"}).status_code == 200 + assert httpx.get(clock.url).json() == {"virtual_time": "2031-02-04T00:00:00.000Z", + "virtual_seconds_per_real_second": 0.0} + assert httpx.get(base + "/clock/state").json()["env_get_time_url"] == clock.url + backwards = httpx.put(base + "/clock/advance", json={"virtual_time": "2031-01-01T00:00:00Z"}) + assert backwards.status_code == 400 and "forward" in backwards.json()["error"] + assert httpx.put(base + "/clock/advance", json={}).status_code == 400 + assert httpx.put(base + "/clock/set-time", content=b"[1]").status_code == 400 + assert httpx.get(base + "/clock/other").status_code == 404 + + +def test_web_is_installed_when_it_has_no_node_modules_and_built_when_dist_is_older_than_a_source(tmp_path): + web = tmp_path / "web" + (web / "src").mkdir(parents=True) + (web / "package.json").write_text("{}") + (web / "src" / "main.js").write_text("") + assert website_dev.web_plan(web, watch=False) == ( + [["npm", "install", "--no-audit", "--no-fund"], ["npm", "run", "build"]], None) + (web / "package-lock.json").write_text("{}") + (web / "node_modules" / "vite").mkdir(parents=True) + (web / "dist").mkdir() + (web / "dist" / "index.html").write_text("") + for path in [*web.rglob("*")]: + os.utime(path, (1000, 1000)) + os.utime(web / "dist" / "index.html", (2000, 2000)) + (web / "node_modules" / "vite" / "new.js").write_text("") + assert not website_dev.web_stale(web) + assert website_dev.web_plan(web, watch=True) == ([], ["npm", "run", "build", "--", "--watch"]) + os.utime(web / "src" / "main.js", (3000, 3000)) + assert website_dev.web_stale(web) + assert website_dev.web_plan(web, watch=False)[0] == [["npm", "run", "build"]] + (web / "node_modules").rename(web / "gone") + assert website_dev.web_plan(web, watch=False)[0][0] == ["npm", "ci", "--no-audit", "--no-fund"] + + +def test_priming_syncs_the_site_to_the_clock_and_loads_the_seed_in_place_of_its_state(site_dir, clock, monkeypatch): + client = TestClient(_app(site_dir, monkeypatch)) + client.post("/api/items", json={"title": "made before priming"}) + seed = site_dir / "data" / "seed.json" + boot = website_dev.prime(client, "http://testserver", clock.url, seed) + assert boot == client.get("/healthz").headers[website_dev.BOOT_HEADER] + assert client.get("/healthz").headers[UNIVERSE_NOW_HEADER].startswith("2030-01-01T09:0") + assert [i["title"] for i in client.get("/api/items").json()] == ["Try the site"] + assert client.post("/api/items", json={"title": "now"}).json()["created_at"].startswith("2030-01-01T09:0") + + +def _card_site(card: dict, sent: list) -> httpx.Client: + """A client of a site that serves ``card`` and records each request it is sent, answering every other one.""" + def answer(request: httpx.Request) -> httpx.Response: + sent.append(request) + if request.url.path == WELL_KNOWN_PATH: + return httpx.Response(200, json=card) + return httpx.Response(200, json={"jsonrpc": "2.0", "id": 1, "result": {}}) + + return httpx.Client(transport=httpx.MockTransport(answer)) + + +def test_priming_loads_the_seed_at_the_data_plane_the_card_names_and_refuses_one_it_doesnt_serve(site_dir): + seed = site_dir / "data" / "seed.json" + clock_ext = {"uri": website_dev.CLOCK_EXTENSION_URI, + "params": {"methods": {"sync_time": {"endpoint": "/agentenv/ext/clock/sync"}}}} + card = {"name": "shop", "url": "/svc/shop/rpc", "capabilities": {"extensions": [clock_ext], + "operations": ["data/reset", "data/add"]}} + sent: list[httpx.Request] = [] + website_dev.prime(_card_site(card, sent), "http://site", "http://clock", seed) + rpcs = [(str(r.url), json.loads(r.content)["method"]) for r in sent + if r.url.path not in (WELL_KNOWN_PATH, "/agentenv/ext/clock/sync")] + assert rpcs == [("http://site/svc/shop/rpc", "data/reset"), ("http://site/svc/shop/rpc", "data/add")] + + sent.clear() + website_dev.prime(_card_site({**card, "url": "http://elsewhere/rpc"}, sent), "http://site", "http://clock", seed) + assert {str(r.url) for r in sent if r.method == "POST" and "clock" not in r.url.path} == {"http://elsewhere/rpc"} + + sent.clear() + unserving = {**card, "capabilities": {**card["capabilities"], "operations": ["data/get"]}} + with pytest.raises(website_dev.DevError, match="doesn't serve data/reset, data/add"): + website_dev.prime(_card_site(unserving, sent), "http://site", "http://clock", seed) + assert not any("rpc" in r.url.path for r in sent) + + +def test_the_supervisor_primes_each_boot_once_and_runs_the_conformance_check_after_the_first(site_dir, clock, + monkeypatch): + first = TestClient(_app(site_dir, monkeypatch)) + said = [] + supervisor = website_dev.Supervisor("http://testserver", clock.url, site_dir / "data" / "seed.json", + echo=said.append, client=first) + with patch.object(website_dev, "prime", wraps=website_dev.prime) as primed: + assert supervisor.step() + assert "[dev] conformance: 9 passed or skipped, 0 failed" in said and primed.call_count == 2 + assert first.get("/healthz").headers[UNIVERSE_NOW_HEADER].startswith("2030-01-01T09:0") + assert supervisor.step() and primed.call_count == 2 + + reloaded = TestClient(_app(site_dir, monkeypatch)) + supervisor.client = reloaded + assert supervisor.step() and primed.call_count == 3 + assert sum("conformance:" in line for line in said) == 1 + assert reloaded.get("/healthz").headers[UNIVERSE_NOW_HEADER].startswith("2030-01-01T09:0") + + +def test_the_supervisor_shows_a_warning_and_syncs_the_site_again_even_when_the_check_raises(site_dir, clock, + monkeypatch): + client = TestClient(_app(site_dir, monkeypatch)) + said = [] + supervisor = website_dev.Supervisor("http://testserver", clock.url, None, echo=said.append, client=client) + warned = conformance.Result("clock", "warning", "interrupted by a reload", False) + with patch.object(conformance, "check", return_value=[warned]), \ + patch.object(website_dev, "prime", wraps=website_dev.prime) as primed: + assert supervisor.step() + assert "[dev] WARN clock: interrupted by a reload" in said and primed.call_count == 2 + + supervisor.checked = False + with patch.object(conformance, "check", side_effect=RuntimeError("boom")), \ + patch.object(website_dev, "prime", wraps=website_dev.prime) as primed, pytest.raises(RuntimeError): + supervisor._conformance() + assert primed.call_count == 1 + + with patch.object(conformance, "check", side_effect=httpx.RemoteProtocolError("reloaded")), \ + patch.object(website_dev, "prime", wraps=website_dev.prime) as primed: + supervisor._conformance() + assert "[dev] the conformance check lost the site, likely to a reload: reloaded" in said and primed.call_count == 1 + + +def test_the_supervisor_primes_again_on_the_next_poll_when_priming_after_the_check_fails(site_dir, clock, + monkeypatch): + client = TestClient(_app(site_dir, monkeypatch)) + said = [] + supervisor = website_dev.Supervisor("http://testserver", clock.url, None, echo=said.append, client=client) + real_prime = website_dev.prime + with patch.object(conformance, "check", return_value=[]), \ + patch.object(website_dev, "prime", side_effect=[real_prime(client, "http://testserver", clock.url, None), + httpx.ConnectError("refused")]): + assert supervisor.step() + assert "[dev] couldn't sync the site again after the check: refused" in said and supervisor.primed is None + with patch.object(website_dev, "prime", wraps=website_dev.prime) as primed: + assert supervisor.step() and primed.call_count == 1 + assert supervisor.primed == client.get("/healthz").headers[website_dev.BOOT_HEADER] + + +def test_the_supervisor_reports_a_seed_it_cant_read_and_primes_once_the_seed_is_back(site_dir, clock, monkeypatch): + client = TestClient(_app(site_dir, monkeypatch)) + said = [] + seed = site_dir / "data" / "seed.json" + kept = seed.read_text(encoding="utf-8") + seed.unlink() + supervisor = website_dev.Supervisor("http://testserver", clock.url, seed, echo=said.append, client=client) + with patch.object(conformance, "check", return_value=[]): + assert supervisor.step() + assert "couldn't sync the site and load its seed" in said[-1] and supervisor.primed is None + + seed.write_text(kept, encoding="utf-8") + real_prime = website_dev.prime + with patch.object(website_dev, "prime", side_effect=[real_prime(client, "http://testserver", clock.url, seed), + FileNotFoundError(str(seed))]): + assert supervisor.step() + assert "couldn't sync the site again after the check" in said[-1] and supervisor.primed is None + + assert supervisor.step() + assert supervisor.primed == client.get("/healthz").headers[website_dev.BOOT_HEADER] + assert [i["title"] for i in client.get("/api/items").json()] == ["Try the site"] + + +def test_the_supervisor_waits_for_a_server_that_isnt_up_and_reports_one_it_cant_prime(): + down = MagicMock(spec=httpx.Client) + down.get.side_effect = httpx.ConnectError("refused") + assert not website_dev.Supervisor("http://127.0.0.1:1", "http://clock", None, client=down).step() + + app = declare(domains=["x.test"])(type("Bare", (website_dev.WebsiteEnvironment,), {}))().create_app() + said = [] + supervisor = website_dev.Supervisor("http://testserver", "http://clock", None, client=TestClient(app), + echo=said.append, check=False) + with patch.object(website_dev, "find_extension_method", return_value=None): + assert supervisor.step() + assert "couldn't sync the site" in said[0] + + +def test_dev_runs_uvicorn_with_reload_beside_the_clock_and_the_supervisor(site_dir, monkeypatch): + monkeypatch.setenv(website_dev.SITE_ENV, "unset") + clock, supervisor = MagicMock(url="http://127.0.0.1:5555/clock/time"), MagicMock() + supervisor.start.return_value = supervisor + clock.start.return_value = clock + with patch.object(website_dev, "DevClock", return_value=clock) as made_clock, \ + patch.object(website_dev, "Supervisor", return_value=supervisor) as made_supervisor, \ + patch.object(website_dev, "npm_available", return_value=True), \ + patch.object(website_dev, "run_npm") as npm, \ + patch("uvicorn.run") as run: + result = CliRunner().invoke(website, ["dev", "--dir", str(site_dir), "--port", "8123", "--no-watch", + "--clock-time", "2030-01-01T09:00:00Z"]) + assert os.environ[website_dev.SITE_ENV] == str(site_dir.resolve() / "site.py") + assert result.exit_code == 0, result.output + assert [c.args[0] for c in npm.call_args_list] == [["npm", "install", "--no-audit", "--no-fund"]] + assert made_clock.call_args.args[0] == parse_time("2030-01-01T09:00:00Z") + assert made_supervisor.call_args.args == ("http://127.0.0.1:8123", clock.url, site_dir.resolve() / "data" / "seed.json") + assert run.call_args.args == ("agent_env.cli.env.website_dev:create_app",) + options = run.call_args.kwargs + assert (options["factory"], options["reload"], options["port"]) == (True, True, 8123) + assert options["reload_dirs"] == [str(site_dir.resolve())] + assert options["reload_includes"] == ["*.py", "*.json", "*.toml"] + assert options["reload_excludes"] == [str(site_dir.resolve() / "web")] + assert "http://127.0.0.1:8123/" in result.output and "/clock/advance" in result.output + supervisor.stop.assert_called_once() + clock.close.assert_called_once() + + +def test_dev_refuses_a_folder_without_its_site_and_a_clock_time_that_isnt_one(tmp_path, site_dir): + missing = CliRunner().invoke(website, ["dev", "--dir", str(tmp_path)]) + assert missing.exit_code == 1 and "no site.py" in missing.output + bad = CliRunner().invoke(website, ["dev", "--dir", str(site_dir), "--clock-time", "soon"]) + assert bad.exit_code == 1 and "not an RFC3339 time" in bad.output + (tmp_path / "site.py").write_text("raise RuntimeError('broken')\n") + broken = CliRunner().invoke(website, ["dev", "--dir", str(tmp_path)]) + assert broken.exit_code == 1 and "broken" in broken.output diff --git a/tst/unit/cli/website_new_cli_test.py b/tst/unit/cli/website_new_cli_test.py new file mode 100644 index 000000000..309407977 --- /dev/null +++ b/tst/unit/cli/website_new_cli_test.py @@ -0,0 +1,254 @@ +"""agent-env env website new: a site like the Notes example, which imports and passes the website/v1 conformance +check as it is written; refused over a non-empty folder without --force; and, from an MCP server's source or a tools +JSON, one stub @api route per tool with a seed collection for each.""" + +import importlib.util +import json +import sys +import uuid +from pathlib import Path + +import pytest +from click.testing import CliRunner +from starlette.testclient import TestClient + +from agent_env.cli.env.website import website +from agent_env.cli.env.website_scaffold import CLIENT_DIR, tools_from_json, tools_from_mcp +from agentenv_protocol.website import SESSION_PATH, WebsiteEnvironment, conformance +from agentenv_protocol.website import website as declare + +MAIL_SERVER = Path(__file__).resolve().parents[3] / "src" / "agent_env" / "examples" / "mail-mcp" / "envs" / "mail" +FILES = {".gitignore", "Dockerfile", "Dockerfile.dockerignore", "README.md", "env.toml", "site.py", "data/seed.json", + "web/index.html", "web/package.json", "web/vite.config.js", "web/src/main.js", "web/src/style.css", + "web/src/site-client.js", "web/src/site-client.d.ts"} +FASTMCP_SERVER = ''' +from typing import Annotated, Literal, Optional +from mcp.server.fastmcp import Context, FastMCP +from pydantic import Field + +mcp = FastMCP("calendar") + + +@mcp.tool(name="list_events") +async def list_events(day: str, limit: int = 10, ctx: Context = None) -> list: + """The events of a day.""" + + +@mcp.tool() +def create_event( + title: Annotated[str, Field(description="What the event is.")], + attendees: list[str], + kind: Literal["meeting", "focus"] = "meeting", + notes: Optional[str] = None, + room: str = Field(None, description="Where."), +) -> dict: + """Add an event.""" + + +def helper(x: int) -> int: + return x +''' + + +def _new(*args): + return CliRunner().invoke(website, ["new", *args]) + + +def _load(site_dir: Path) -> type[WebsiteEnvironment]: + """The generated ``site.py``'s site, serving a stand-in page instead of ``web/dist``, which only a Vite build makes.""" + spec = importlib.util.spec_from_file_location(f"generated_site_{uuid.uuid4().hex}", site_dir / "site.py") + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + cls = next(v for v in vars(module).values() if isinstance(v, type) and issubclass(v, WebsiteEnvironment) + and v is not WebsiteEnvironment) + page = site_dir / "stand-in" + page.mkdir() + (page / "index.html").write_text("site") + declared = cls.declaration()._replace(static=str(page), data=str(site_dir / "data" / "seed.json")) + return declare(**declared._asdict())(type(cls.__name__, (cls,), {})) + + +def _passes_conformance(client: TestClient) -> None: + """The conformance check, whose clock check serves a clock on 127.0.0.1 for the site to read.""" + results = conformance.check("http://testserver", client=client) + assert {r.name: r.status for r in results} == dict.fromkeys( + ["card", "any_host", "health", "universe_now", "reserved_paths", "data_plane", "clock", "forwarded", "cookies"], "passed"), results + + +def test_a_new_site_has_notes_layout_and_the_sdks_browser_client(tmp_path): + result = _new("shop", "--dir", str(tmp_path / "shop"), "--domain", "shop.example") + assert result.exit_code == 0, result.output + assert "agent-env env website dev" in result.output and "--health-path" not in result.output + site = tmp_path / "shop" + assert {str(p.relative_to(site)) for p in site.rglob("*") if p.is_file()} == FILES + assert (site / "env.toml").read_text() == ('type = "website"\nenvironment_name = "shop"\n' + 'backend_image = { dockerfile = "Dockerfile" }\n') + assert "--health-path" not in (site / "README.md").read_text() + package = json.loads((site / "web" / "package.json").read_text()) + assert package["devDependencies"] == {"vite": "5.4.10"} and "dependencies" not in package + vite = (site / "web" / "vite.config.js").read_text() + assert 'base: "./"' in vite and '"/api":' in vite and '"/healthz":' in vite + assert 'from "./site-client.js"' in (site / "web" / "src" / "main.js").read_text() + page = (site / "web" / "index.html").read_text() + assert page.index('location.pathname.endsWith("/")') < page.index('