From 64e0b67a614ba9f797c1bca1668dd08b67a987aa Mon Sep 17 00:00:00 2001 From: DVIDIA Date: Fri, 9 Oct 2026 05:55:54 -0400 Subject: [PATCH 1/2] Add Skillspace coverage planner and simulator episode tapes --- .github/workflows/ci.yml | 4 +- README.md | 10 +- docs/episode-tape.md | 119 ++++++ docs/pipeline.md | 8 + docs/roadmap.md | 101 +++++ docs/skillspace-coverage.md | 128 ++++++ examples/coverage-plan.json | 167 ++++++++ pyproject.toml | 2 + src/dvidia_training/coverage.py | 387 +++++++++++++++++ src/dvidia_training/coverage_studio.html | 53 +++ src/dvidia_training/episode_tape.py | 506 +++++++++++++++++++++++ src/dvidia_training/training_studio.html | 2 +- src/dvidia_training/training_studio.py | 9 + tests/test_coverage.py | 235 +++++++++++ tests/test_episode_tape.py | 357 ++++++++++++++++ tests/test_training_studio.py | 144 ++++++- 16 files changed, 2228 insertions(+), 4 deletions(-) create mode 100644 docs/episode-tape.md create mode 100644 docs/roadmap.md create mode 100644 docs/skillspace-coverage.md create mode 100644 examples/coverage-plan.json create mode 100644 src/dvidia_training/coverage.py create mode 100644 src/dvidia_training/coverage_studio.html create mode 100644 src/dvidia_training/episode_tape.py create mode 100644 tests/test_coverage.py create mode 100644 tests/test_episode_tape.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7654be9..f9e0d1e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -59,9 +59,11 @@ jobs: dvidia-train --help dvidia-training-studio --help dvidia-training-benchmark --help + dvidia-skillspace-coverage --help + dvidia-episode-tape --help dvidia-training-doctor --json - name: Test footage, visual training and loopback studio - run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor -v + run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape -v arm: name: Optional arm / Linux / Python 3.12 diff --git a/README.md b/README.md index 68c7bc3..5b094d9 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ # DVIDIA Training -[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md) +[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Execution map](docs/roadmap.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md) [Recorded demo](https://huggingface.co/spaces/Dvidia/dvidia-training) · [Synthetic datasets](https://huggingface.co/datasets/Dvidia/dvidia-training-examples) · [Pilot models](https://huggingface.co/Dvidia/dvidia-training-pilot) @@ -32,6 +32,14 @@ Open [http://127.0.0.1:8270](http://127.0.0.1:8270). Select a local Skillspace f For Windows, use `.venv\Scripts\python.exe` and `.venv\Scripts\dvidia-training-studio.exe`; [complete PowerShell commands](docs/installation.md#windows-powershell) are provided. +## Plan the next useful recording + +Open [the collection planner](http://127.0.0.1:8270/coverage) in the same local studio. Define the skill's goal, operating envelope, selected conditions and simulation benchmark. The planner suggests a capture brief for a gap and tallies declared independent reviewed groups. Its example records are fabricated; it does not inspect footage or establish training readiness. Export the plan to keep it. + +[Collection design and proposed incentives](docs/skillspace-coverage.md) explain complete attempts, review and rights declarations, source grouping, failures and recoveries. Every skill starts with a benchmark description; an unsupported task environment keeps qualification blocked. Collection targets and measured competence are separate. + +The [episode tape recorder](docs/episode-tape.md) preserves the existing simulated arm's measured state, requested/applied commands, contacts and sensor ages. Recording needs the optional arm extra; validation and inspection need only Python's standard library. These tools are available from the current checkout and do not train a new policy or change the older Hugging Face pilot. + ## Try the pipeline Generate an explicitly synthetic example, then train and verify its output: diff --git a/docs/episode-tape.md b/docs/episode-tape.md new file mode 100644 index 0000000..198d23a --- /dev/null +++ b/docs/episode-tape.md @@ -0,0 +1,119 @@ +# Simulator episode tapes + +Episode tapes preserve what the authored research arm observed, requested and +actually did at each control boundary. They are a separate versioned evidence +format. They do not change the existing movement-training sidecar, train a +supervisor, recover actions from human videos or qualify physical hardware. + +The recorder runs locally and offline with the optional `arm` dependencies. +Validation and inspection use only the Python standard library. No renderer, +GPU, FFmpeg, account, API key or network is needed for these operations after +installation. These are dependency requirements; minimum RAM and throughput +have not been established across machines. + +## Record fresh evidence + +Install the simulator extra using the [installation guide](installation.md). +Use a fresh output path outside the frozen benchmark directories: + +```sh +mkdir -p runs/episode-tapes +python -m dvidia_training.episode_tape record \ + --output runs/episode-tapes/nominal-19050.json \ + --episode-id nominal-19050 \ + --source-recording-id recording-nominal-19050 \ + --session-id simulation-session-19050 \ + --seed 19050 --case nominal + +python -m dvidia_training.episode_tape record \ + --output runs/episode-tapes/open-jaw-19050.json \ + --episode-id open-jaw-19050 \ + --source-recording-id recording-open-jaw-19050 \ + --session-id simulation-session-19050 \ + --seed 19050 --case open-jaw + +python -m dvidia_training.episode_tape validate runs/episode-tapes/nominal-19050.json +python -m dvidia_training.episode_tape inspect runs/episode-tapes/open-jaw-19050.json +``` + +The `nominal` case records the unchanged authored controller. `open-jaw` keeps +its joint requests while issuing an 80 mm jaw gap. `release-after-lift` requests +an 80 mm jaw gap at control boundaries falling within 0.35 simulation seconds +after the environment first reports a retained lift. With the default 20 ms +control interval, the final such request also governs the remainder of its +step, so the overridden command intervals span 0.36 seconds. The native target +ramp still limits opening speed, and measured aperture may lag the requested +gap. The recorder then returns command authority to the unchanged authored controller. The +interventions change jaw commands and leave native object dynamics intact. +Their outcomes depend on the authored scene and controller; these examples +are recording checks, not a new generalization benchmark. + +Related recordings must stay in the same source/session split group. The two +examples above share a session because they reuse the same seeded scene. +Assign stable IDs when collecting evidence; changing an ID cannot make a +duplicate or counterfactual recording independent. The recorder preserves +these IDs but does not assign dataset splits or inspect an existing corpus. + +Output creation refuses to overwrite an existing file. A malformed tape is +rejected before a write. A broken native clock or invalid numeric observation +cannot be exported as valid evidence. Inspection never modifies the input. + +## What the v1 format preserves + +| Component | Meaning | +| --- | --- | +| Provenance | Episode, recording and session IDs; seed and case; authored policy revision; MuJoCo version; scene, configuration and source-file hashes. | +| Clock | Simulation elapsed seconds, with zero after reset settling and a native-clock offset. Wall-clock timing and hardware clock synchronization are outside v1. | +| Initial / before / after | Measured simulator state at zero and at each control boundary. The previous after-state must equal the next before-state. | +| Policy request | Authored joint targets in radians and requested jaw gap in metres. | +| Issued command | The command passed to the environment, separately retaining any declared jaw intervention. Issue time equals the before-state time. | +| Applied command | The step’s start/end times, final ramped targets, final native servo controls and rate-limit flag. These are commands, not measured positions or velocities. | +| Measurements | Joint positions/velocities/actuator torques; jaw actuator forces and measured gap; per-pad normal forces and object contacts; table contacts; tool and object pose/velocity; target position. | +| Authored labels | Action phase and next phase captured after policy evaluation, before the step. These are controller evidence, not approved human labels or deployment inputs. | +| Outcome | Terminal reason, success, validity, simulation duration, step count, authored failure reason and recovery count. These must agree with the final transition. | + +Each signal declares its dimensions, units and coordinate frame. Quaternions +use `w, x, y, z`. All recorded signals come from privileged simulator state; +the format does not claim that a physical arm has equivalent sensors. Contact +normal force in newtons is distinct from jaw actuator force. Requested closing +and a measured gap alone are not labels for a successful grasp. + +A signal is `available`, `stale` or `missing`. Available and stale readings +retain acquisition time and age. The v1 freshness limit is 0.1 seconds, a +recording-contract choice rather than a validated hardware deadline. Missing +readings have null value, acquisition time and age. They are never fabricated +as zeros. Current native recordings acquire all available signals at the +observation boundary, so their age is zero; stale/missing support is validated +for future data producers but no sensor-fault simulator is added here. + +The tape samples at the environment’s control interval, ordinarily 20 ms. It +does not retain the intermediate 1 ms native ticks. Final servo controls +include the environment’s gravity compensation. Applied-command timestamps +describe the entire transition and the final controls, not a claim that the +final command was active throughout that interval. Intra-step force peaks and +short contacts can therefore be missed by this tape. + +Validation refuses unsupported versions/scopes, unknown or duplicate fields, +nonfinite/boolean numeric entries, malformed shapes or quaternions, inconsistent +sensor ages, contradictory missingness, reordered or discontinuous observations, +command/transition timing mismatches, unsupported actuator commands, inconsistent +terminal outcomes and changed protocol/configuration hashes. Limits are 10,000 +control steps, one million native ticks per control or settling interval, and +64 MiB per tape. The clock origin must match the declared native settling ticks, +and both native jaw controls must equal half the final ramped gap. Hashes +establish consistency, not provenance +authenticity, sensor accuracy or task mastery. + +## Next consumers + +The first consumers should be a synchronized episode inspector and a frozen +training/confirmation split, followed by a compact state-based outcome model. +The current visual predictor and movement head do not consume these tapes. +A future learner must choose deployable inputs explicitly and hold privileged +object state and authored phase labels out of its deployment feature vector. + +Every eventual skill candidate needs its own versioned simulator task, +observable success/failure conditions, fresh held-out scenarios, matched controls, +complete attempted-episode receipts, performance measurements and a supported +robot/gripper envelope. A validated tape is an input to that process; it is not +an installable qualified skill. diff --git a/docs/pipeline.md b/docs/pipeline.md index f37de6b..6349d8a 100644 --- a/docs/pipeline.md +++ b/docs/pipeline.md @@ -10,6 +10,7 @@ The product path is: ```mermaid flowchart LR + P[Plan skill coverage and simulation benchmark] --> A A[Skillspace folder, ZIP or manifest link] --> B[Check actual videos and source groups] B --> C[Fix training, development and test splits] C --> D[Learn a visual model offline] @@ -42,6 +43,13 @@ check the footage, then train and download the result. The server binds only to your own machine. To use it on your server, keep that binding and forward the port through SSH. Do not expose this local file-reading interface publicly. +Before collecting, open `/coverage` to define the operating envelope, selected +capture cases and each skill's benchmark description. The [coverage planner](skillspace-coverage.md) +tallies supplied review metadata and exports a companion plan; it does not inspect +videos or run training. See the [execution map](roadmap.md) for the remaining gates. +The [episode tape recorder](episode-tape.md) separately preserves simulated state +and commands for future outcome/phase models; current learners do not consume it. + The default server prohibits downloads. `--allow-online` enables an explicit per-job option to fetch a public DVIDIA manifest and its declared media. Training itself remains local. A catalog URL with a claimed demonstration count but no diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..2970115 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,101 @@ +# Execution map: footage to installable skills + +Working map, 9 October 2026. This is the team's sequence of builds and evidence +gates. A completed software component does not establish robot competence. + +## The product loop + +```mermaid +flowchart LR + A[Define skill goal and operating envelope] --> B[Specify its simulation benchmark] + B --> C[Request recordings that fill coverage gaps] + C --> D[Review footage, rights, outcomes and source groups] + D --> E[Freeze development and confirmation groups] + E --> F[Train compatible candidates locally] + F --> G[Evaluate every attempted simulation trial] + G --> H{Meets the declared task criteria?} + H -->|No| I[Diagnose development failures and request new evidence] + I --> C + H -->|Yes| J[Publish versioned candidate, compatibility and results] + J --> K[Install through a compatible adapter and calibrate] + K --> L[Separate physical robot qualification] +``` + +Confirmation evidence is opened after a candidate is frozen. Once inspected, +that suite becomes regression evidence for later changes. Further qualification +needs fresh confirmation scenarios; it cannot silently reuse a known test set. + +## Current foundation and next builds + +| Order | Build and owner role | Current state | Evidence needed to advance | +| --- | --- | --- | --- | +| 1 | Collection planning — product + data | Local coverage planner, versioned briefs and declared source-group tallies implemented. | First skill's observable goal, operating envelope, actual recordings and reviewed capture lineage. | +| 2 | Contribution review — data + product | Proposed official-site connection; the local planner imports declarations only. | Immutable review receipts tied to original bytes, plan revision, intended-use permission and complete outcomes; source-group and near-duplicate audits. | +| 3 | Episode evidence — simulation | Recorder preserves state, requests, applied commands, sensor age and authored phases at control boundaries. | Task-specific episode corpus, retained failure cases and synchronized inspector. Tapes are not consumed by a learner yet. | +| 4 | Task and state models — learning | Existing small visual predictor and bounded movement head; outcome/phase supervisor proposed. | Choose deployable sensor inputs, compare with simple baselines and measure held-out boundary, outcome and false-completion errors. Keep authored labels and privileged state outside deployment inputs. | +| 5 | Task simulation — simulation | Existing rigid-box placement adapter and frozen benchmark; other tasks need environments. | Validated assets/contact behavior, robot/gripper/sensor contract, reset distributions and independently observable task outcomes. | +| 6 | Qualification and sample efficiency — evaluation | Local benchmark runner and frozen sample-count evidence exist. | Learning curves over independent source groups, multiple collection orders, frozen candidates, nominal/stress/control strata, all attempted trials and measured cost. | +| 7 | Installable skill — runtime | Simulation candidate capsule and compatibility checks exist. | Bundle model, task, observation/action contract, adapter, data/protocol hashes, calibration requirements and artifact-bound qualification results. | +| 8 | Physical transfer — robotics | Unqualified. | Hardware timing/calibration checks, measured limits and a separate supervised physical evaluation. Simulation results alone cannot advance this stage. | + +The coverage planner and recorder are available from the current source checkout. +They are not included in the frozen v0.1.0 release or the older Hugging Face pilot. +No new task policy is trained by adding these components. + +## First incoming skills + +Start with one bounded task and one supported robot/gripper rather than declaring +a universal skill. For shoe organization, distinguish pairing, alignment and +placing on a shelf: each needs a goal, observable result and task environment. +Record initial conditions, the complete attempt and its outcome. Five- or +six-second clips are useful when they contain that whole transition. Preserve +longer originals and shared recording/session/object IDs across every cut. + +Collection starts with selected high-value conditions: different physical +instances, starting poses, views, clutter, failed grasps and recoveries. Initial +quotas are workload estimates. Use development errors to refine the plan; +version any changed brief and re-review earlier evidence against it. Freeze +confirmation groups before fitting or adjusting the policy. + +Human footage can support visual representations and task labels. Joint commands, +forces and calibrated geometry require robot telemetry or a validated action +bridge. Synthetic variations preserve their parent lineage and are reported +separately from independently captured evidence. Shoe tying needs a deformable +task environment; it cannot inherit the rigid-box adapter's qualification. + +## What “enough data” means + +Report independent groups, conditions and outcomes, not just video count or +seconds. Run nested training subsets while keeping development evidence fixed. +Predeclare the target success, false-completion and critical-error limits with +uncertainty estimates and a cost budget. Freeze the selected candidate before +opening fresh confirmation trials. A plateau below the target calls for a +diagnosis of observations, actions, model or simulator, not a “done” badge. + +Collection coverage, usable data, trained candidate, simulation qualification and +physical qualification remain separate states. A collection quota cannot promote +any of the later states. + +## Every skill's benchmark contract + +At creation, record a success predicate, meaningful failure cases and whether an +environment exists. A missing environment blocks qualification while collection +and environment work continue. The planner's description is the start of the +contract; an executable benchmark additionally needs: + +- Task/environment, asset, physics, controller and sensor versions; observation + and action units, coordinate frames, degrees of freedom and gripper limits. +- Reset distributions, seeds, episode timeout and nominal, stress, recoverable, + impossible-task and sensor-fault strata where applicable. +- Authored/simple baselines and matched negative controls; a policy hash and + frozen dataset/protocol hashes before evaluation. +- All attempted episodes, including failures, errors and incompatibilities; + success, false completion, recovery and critical-error rates with denominators. +- Wall time, step throughput, end-to-end control latency and memory with named + hardware and measurement boundaries. CPU-only support is measured per task. +- A supported operating envelope and explicit simulation-only or physical scope. + +The next engineering milestone is a tape inspector and a small state-based task +outcome model on the existing placement environment. The first real Skillspace +will then test the collection and review loop. See [collection design](skillspace-coverage.md), +[episode evidence](episode-tape.md) and [existing benchmark protocol](benchmarks.md). diff --git a/docs/skillspace-coverage.md b/docs/skillspace-coverage.md new file mode 100644 index 0000000..aee09a6 --- /dev/null +++ b/docs/skillspace-coverage.md @@ -0,0 +1,128 @@ +# Collect evidence that covers a Skillspace + +“Exhausting” a skill means covering a declared operating envelope and checking +its remaining gaps. It cannot mean enumerating every future object or situation. +This guide separates the implemented local planning tool from proposed public +contribution and incentive features. + +## Use the local planner + +Install from the current checkout, then start `dvidia-training-studio` as described +in [installation](installation.md). Open +[http://127.0.0.1:8270/coverage](http://127.0.0.1:8270/coverage). The page offers an +explicitly fabricated six-case example, a fresh-plan button, editable goals and +conditions, next-recording briefs, JSON import/export and a benchmark description. +It uses no account, GPU, external model or network after installation. + +The page keeps its plan in memory. Export it to preserve work; a refresh loads the +example again. It does not upload, inspect or approve footage, run training or +execute benchmarks. The existing footage checker remains a separate step. + +For the command line, the planner itself needs only Python's standard library: + +```sh +.venv/bin/dvidia-skillspace-coverage example --output skillspace.coverage.json +.venv/bin/dvidia-skillspace-coverage inspect skillspace.coverage.json +``` + +The companion plan is separate from `skillspace.training.json`. Its exact v1 +schema is illustrated in [examples/coverage-plan.json](../examples/coverage-plan.json) +and validated by `dvidia_training.coverage.validate_plan`. The command line and +the coverage endpoint accept at most 1 MiB. Training requests retain their 32 KiB limit. +No paths or executable code are resolved from this plan. + +Each plan has a stable ID, revision, goal, operating envelope, selected cases and +benchmark description. Each case has a stable ID, specific capture brief, +requested outcome, group quota and priority. Unknown outcomes can be recorded, +but a planning cell requesting an unknown outcome remains unresolved. + +## What counts toward a collection target + +The tool counts declared evidence only when it is complete, has intended-use +rights approved, has an accepted human review for the current revision, matches +a known requested outcome and belongs to training or development evidence. +Recording, session and physical object IDs must all be present. Evidence also +binds the media SHA-256 and case ID. Proposed/AI-only reviews, stale reviews, +partial attempts, missing lineage and permission gaps remain visible as exclusions. + +Media hashes and recording/session/object IDs connect source groups transitively. +Multiple cuts from one source contribute one group to a cell. A group touching +confirmation evidence or mixing training and development is excluded as a whole. +Conflicting accepted outcomes for identical media cannot be cherry-picked. +Different IDs cannot prove independence; reviewers still need original footage +and near-duplicate checks. These declarations are not authenticated by this tool. + +Changing the plan creates a new revision. Earlier records retain their history; +they need review against the new requirements before they satisfy the new targets. +The planner always reports benchmark execution/qualification, acquired robot +skill and physical readiness as false. A descriptor cannot serve as a result. + +## Proposed contribution experience + +Use a specific mission rather than a generic upload request. For example: +“Record a complete alignment attempt with a different physical pair, starting +sideways.” Explain which coverage gap it addresses and show a short example, +capture checklist and optional practice step. Allow a contributor to choose a +different feasible mission. Do not force a five-second limit that hides an outcome. + +After intake, show received, checked, reviewed and accepted/correction-needed +states. A review receipt should name the plan revision, recording hash, outcome, +permission and group decision, with a useful reason for corrections. Acceptance +and verified diversity require the future review service; they are not supplied +by this prototype. Protect private originals and make public credit opt-in. + +The next mission should prioritize an unfilled critical condition or a diagnosed +development failure. Avoid a full Cartesian product of all dimensions. Choose +task-relevant interactions among object instances, start/goal configurations, +views, lighting, clutter, outcomes and robot/gripper capabilities. + +This draws on DROID's task prompts, preferred-task tracking, practice mode and +periodic scene-change requests. Camera changes also require recalibration in its +workflow. [DROID collection documentation](https://droid-dataset.github.io/droid/example-workflows/data-collection.html). +LIBERO separates spatial, object and goal variations and supplies RGB, +proprioception and scene/task descriptions; those distinctions inform our case +design, not a claim of equivalent task coverage. +[LIBERO datasets](https://libero-project.github.io/datasets). + +## Incentives to test + +Prefer team coverage milestones and opt-in credit for accepted independent +evidence, difficult conditions, honest failures and useful review corrections. +Avoid leaderboards based on raw upload count, clip length or successes alone: +they would encourage duplicate cuts and concealed failures. + +For a future funded campaign, publish the budget, eligible missions, acceptance +terms and review process before recording. A fair base payment for compliant +complete attempts plus a bounded bonus for underrepresented conditions is a +design hypothesis. An unsuccessful task attempt can still be valuable evidence. +This build creates no funded campaign, points, payment or public attribution. + +RoboCrowd studied material rewards, intrinsic interest and social comparison in +in-person robot teleoperation. Its engagement and policy results make incentive +design worth testing here; they do not establish the best incentive for remote +Skillspace contributors, and mixing crowd data can affect policy quality. +[RoboCrowd paper](https://arxiv.org/html/2411.01915v2). + +Test a mission-based flow against a generic upload flow with equal budgets and +participant-level assignment. Measure gaps closed, independently reviewed +groups per contributor-hour, audited label errors, complete attempts, satisfaction +and return participation. Include assigned people who never submit in engagement +denominators. No effectiveness result has been measured for DVIDIA's proposed UX. + +## Enough to train, then enough to qualify + +Initial quotas guide collection effort. They do not determine data sufficiency. +Evaluate learning curves across independent groups and collection orders; use +development evidence to diagnose weak conditions. Preserve frozen confirmation +groups and report success, false completion and critical errors over all attempted +trials. Every skill needs its own simulator task and compatibility envelope. + +Synthetic augmentation may increase useful variation once its task adapter is +validated, but it remains related to its parent recordings. MimicGen's custom +environment interface requires object poses, subtask boundaries and action/pose +conversions; it is not an arbitrary human-video-to-action converter. +[MimicGen custom environment documentation](https://mimicgen.github.io/docs/tutorials/datagen_custom.html). + +The [execution map](roadmap.md) lists the separate collection, review, learning, +benchmark and installation gates. The existing placement adapter cannot establish +shoe organization or deformable lace-tying competence. diff --git a/examples/coverage-plan.json b/examples/coverage-plan.json new file mode 100644 index 0000000..6b334f9 --- /dev/null +++ b/examples/coverage-plan.json @@ -0,0 +1,167 @@ +{ + "format": "dvidia.skillspace-coverage-plan", + "schema_version": 1, + "plan_id": "demo-rigid-placement", + "plan_revision": 1, + "skill": { + "name": "Rigid object placement · planning example", + "goal": "Move one supported rigid box onto a marked target and verify stable release.", + "operating_envelope": "Authored six-DOF arm with parallel jaws and bounded rigid-box table scene. This is a capture-planning example; shoes, deformables and physical hardware are outside the current adapter." + }, + "cases": [ + { + "id": "nominal", + "label": "Nominal placement", + "brief": "Show one complete rigid-object placement, from initial state through release and stable final position.", + "outcome": "success", + "required_groups": 2, + "priority": 6 + }, + { + "id": "new-instance", + "label": "Different physical object", + "brief": "Use another physical instance within the supported size and material envelope. Preserve its stable object identity.", + "outcome": "success", + "required_groups": 2, + "priority": 8 + }, + { + "id": "empty-grasp", + "label": "Empty grasp", + "brief": "Record an observed empty closure with the jaws and object visible, preserving the failed outcome. Do not label commanded closure as holding.", + "outcome": "failure", + "required_groups": 1, + "priority": 9 + }, + { + "id": "recovery", + "label": "Recovery after missed grasp", + "brief": "Capture the missed grasp, corrective attempt and final observed result in one complete episode.", + "outcome": "recovery", + "required_groups": 2, + "priority": 10 + }, + { + "id": "occlusion", + "label": "Brief visibility loss", + "brief": "Capture a complete placement with a documented short visibility interruption and observable final state. Retain uncertainty where contact is not visible.", + "outcome": "success", + "required_groups": 2, + "priority": 7 + }, + { + "id": "rotated-start", + "label": "Rotated starting object", + "brief": "Vary the initial object orientation within the supported envelope, keeping the whole attempt and final outcome visible.", + "outcome": "success", + "required_groups": 2, + "priority": 5 + } + ], + "benchmark": { + "status": "specified", + "environment": "Proposed contact-pick-place-box-v0 simulation suite; no new suite is executed by this planner.", + "reason": "Descriptor only. Fresh frozen scenarios and artifact-bound results are still required.", + "success_predicate": "Observe grasp, lift, transfer, release and target stability using the independent native task gate; count every attempted trial.", + "failure_cases": [ + "Empty grasp", + "Slip or drop", + "Unreachable start", + "Insufficient jaw force", + "Observation loss", + "Matched open-jaw negative control" + ] + }, + "evidence": [ + { + "id": "demo-nominal-a", + "plan_revision": 1, + "case_id": "nominal", + "media_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "complete": true, + "outcome": "success", + "review_state": "accepted", + "review_origin": "human", + "rights_status": "approved", + "split": "train", + "source_recording_id": "demo-recording-nominal-a", + "session_id": "demo-session-nominal-a", + "object_instance_id": "demo-object-nominal-a" + }, + { + "id": "demo-nominal-b", + "plan_revision": 1, + "case_id": "nominal", + "media_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "complete": true, + "outcome": "success", + "review_state": "accepted", + "review_origin": "human", + "rights_status": "approved", + "split": "train", + "source_recording_id": "demo-recording-nominal-b", + "session_id": "demo-session-nominal-b", + "object_instance_id": "demo-object-nominal-b" + }, + { + "id": "demo-instance-a", + "plan_revision": 1, + "case_id": "new-instance", + "media_sha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "complete": true, + "outcome": "success", + "review_state": "accepted", + "review_origin": "human", + "rights_status": "approved", + "split": "dev", + "source_recording_id": "demo-recording-instance-a", + "session_id": "demo-session-instance-a", + "object_instance_id": "demo-object-instance-a" + }, + { + "id": "demo-failure-a", + "plan_revision": 1, + "case_id": "empty-grasp", + "media_sha256": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "complete": true, + "outcome": "failure", + "review_state": "accepted", + "review_origin": "human", + "rights_status": "approved", + "split": "train", + "source_recording_id": "demo-recording-failure-a", + "session_id": "demo-session-failure-a", + "object_instance_id": "demo-object-failure-a" + }, + { + "id": "demo-recovery-proposal", + "plan_revision": 1, + "case_id": "recovery", + "media_sha256": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee", + "complete": true, + "outcome": "recovery", + "review_state": "proposed", + "review_origin": "ai", + "rights_status": "approved", + "split": "train", + "source_recording_id": "demo-recording-recovery-proposal", + "session_id": "demo-session-recovery-proposal", + "object_instance_id": "demo-object-recovery-proposal" + }, + { + "id": "demo-occlusion-pending", + "plan_revision": 1, + "case_id": "occlusion", + "media_sha256": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "complete": true, + "outcome": "success", + "review_state": "proposed", + "review_origin": "human", + "rights_status": "unknown", + "split": "train", + "source_recording_id": "demo-recording-occlusion-pending", + "session_id": "demo-session-occlusion-pending", + "object_instance_id": "demo-object-occlusion-pending" + } + ] +} diff --git a/pyproject.toml b/pyproject.toml index 29ac577..3d6c0d8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -30,6 +30,8 @@ dvidia-training-studio = "dvidia_training.training_studio:main" dvidia-training-doctor = "dvidia_training.doctor:main" dvidia-training-benchmark = "dvidia_training.footage_benchmark:main" dvidia-skill-capsule = "dvidia_training.arm_entrypoints:capsule_main" +dvidia-skillspace-coverage = "dvidia_training.coverage:main" +dvidia-episode-tape = "dvidia_training.episode_tape:main" [tool.setuptools.packages.find] where = ["src"] diff --git a/src/dvidia_training/coverage.py b/src/dvidia_training/coverage.py new file mode 100644 index 0000000..0a5bf80 --- /dev/null +++ b/src/dvidia_training/coverage.py @@ -0,0 +1,387 @@ +"""Bounded, data-only capture planning; declarations never establish robot ability. + +This companion contract does not change the footage manifest, run models, inspect +media, award rewards, or execute a benchmark. Source identities and review/rights +statements are declarations whose truth must be established outside this tool. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path +import re +import sys + + +FORMAT = 'dvidia.skillspace-coverage-plan' +SCHEMA_VERSION = 1 +MAX_PLAN_BYTES = 1024 * 1024 +MAX_CASES = 64 +MAX_EVIDENCE = 1200 +OUTCOMES = ('success', 'failure', 'recovery', 'unknown') +REVIEW_STATES = ('accepted', 'proposed', 'rejected', 'needs_changes', 'stale') +PLAN_FIELDS = {'format', 'schema_version', 'plan_id', 'plan_revision', 'skill', + 'cases', 'benchmark', 'evidence'} +CASE_FIELDS = {'id', 'label', 'brief', 'outcome', 'required_groups', 'priority'} +EVIDENCE_FIELDS = {'id', 'plan_revision', 'case_id', 'media_sha256', + 'source_recording_id', 'session_id', 'object_instance_id', + 'outcome', 'complete', 'review_state', 'review_origin', + 'rights_status', 'split'} +LINEAGE_FIELDS = ('source_recording_id', 'session_id', 'object_instance_id') + + +class CoverageError(ValueError): + """A plan cannot be interpreted without relaxing its declared contract.""" + + +def _canonical(value): + try: + return json.dumps(value, sort_keys=True, separators=(',', ':'), + ensure_ascii=False, allow_nan=False).encode('utf-8') + except (ValueError, TypeError, UnicodeError, RecursionError) as exc: + raise CoverageError('Plan must be finite UTF-8 JSON.') from exc + + +def _read(value): + if type(value) is dict: + raw = _canonical(value) + document = value + elif type(value) in (str, bytes): + try: + raw = value.encode('utf-8') if type(value) is str else value + except UnicodeError as exc: + raise CoverageError('Plan must be valid UTF-8 JSON.') from exc + if len(raw) > MAX_PLAN_BYTES: + raise CoverageError('Plan exceeds the 1 MiB JSON limit.') + + def pairs(items): + result = {} + for key, item in items: + if key in result: + raise CoverageError('Duplicate JSON keys are not allowed.') + result[key] = item + return result + + def constant(_): + raise CoverageError('Nonfinite JSON values are not allowed.') + + try: + document = json.loads(raw.decode('utf-8'), object_pairs_hook=pairs, + parse_constant=constant) + except (ValueError, UnicodeError, RecursionError) as exc: + if isinstance(exc, CoverageError): + raise + raise CoverageError('Plan must be valid finite UTF-8 JSON.') from exc + _canonical(document) + else: + raise CoverageError('Expected a plain object, JSON text, or JSON bytes.') + if len(raw) > MAX_PLAN_BYTES: + raise CoverageError('Plan exceeds the 1 MiB JSON limit.') + return document + + +def _record(value, fields, path): + if type(value) is not dict or set(value) != fields: + raise CoverageError(path + ': expected exactly the documented fields.') + return value + + +def _text(value, path, maximum=1600, *, blank=False): + if type(value) is not str or len(value) > maximum or (not blank and not value.strip()): + raise CoverageError(path + ': expected bounded text' + (' or an empty string.' if blank else '.')) + if any((ord(c) < 32 and c not in '\t\n\r') or 127 <= ord(c) <= 159 + or 0xD800 <= ord(c) <= 0xDFFF for c in value): + raise CoverageError(path + ': unsupported control characters or Unicode.') + return value.strip() + + +def _identity(value, path): + if type(value) is not str or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.:-]{0,95}', value): + raise CoverageError(path + ': expected a stable identifier of at most 96 characters.') + return value + + +def _integer(value, path, minimum, maximum): + if type(value) is not int or not minimum <= value <= maximum: + raise CoverageError(path + f': expected an integer in [{minimum}, {maximum}].') + return value + + +def _choice(value, choices, path): + if type(value) is not str or value not in choices: + raise CoverageError(path + ': unsupported value.') + return value + + +def validate_plan(plan): + """Return a detached normalized contract, rejecting invented qualification.""" + row = _record(_read(plan), PLAN_FIELDS, 'plan') + if row['format'] != FORMAT or type(row['schema_version']) is not int or row['schema_version'] != 1: + raise CoverageError('Unsupported coverage plan format or schema version.') + identity = _identity(row['plan_id'], 'plan_id') + revision = _integer(row['plan_revision'], 'plan_revision', 1, 1_000_000) + skill = _record(row['skill'], {'name', 'goal', 'operating_envelope'}, 'skill') + skill = {'name': _text(skill['name'], 'skill.name', 120), + 'goal': _text(skill['goal'], 'skill.goal'), + 'operating_envelope': _text(skill['operating_envelope'], 'skill.operating_envelope')} + if type(row['cases']) is not list or not 1 <= len(row['cases']) <= MAX_CASES: + raise CoverageError('cases: expected 1–64 planned cases.') + cases, case_ids = [], set() + for index, item in enumerate(row['cases']): + path = f'cases[{index}]' + item = _record(item, CASE_FIELDS, path) + case_id = _identity(item['id'], path + '.id') + if case_id in case_ids: + raise CoverageError('Case identities must be unique.') + case_ids.add(case_id) + cases.append({'id': case_id, 'label': _text(item['label'], path + '.label', 180), + 'brief': _text(item['brief'], path + '.brief'), + 'outcome': _choice(item['outcome'], OUTCOMES, path + '.outcome'), + 'required_groups': _integer(item['required_groups'], path + '.required_groups', 1, 1000), + 'priority': _integer(item['priority'], path + '.priority', 1, 10)}) + benchmark = _record(row['benchmark'], {'status', 'environment', 'reason', + 'success_predicate', 'failure_cases'}, 'benchmark') + benchmark = {'status': _choice(benchmark['status'], ('blocked', 'specified'), 'benchmark.status'), + 'environment': _text(benchmark['environment'], 'benchmark.environment', blank=True), + 'reason': _text(benchmark['reason'], 'benchmark.reason', blank=True), + 'success_predicate': _text(benchmark['success_predicate'], 'benchmark.success_predicate', blank=True), + 'failure_cases': benchmark['failure_cases']} + if type(benchmark['failure_cases']) is not list or len(benchmark['failure_cases']) > 64: + raise CoverageError('benchmark.failure_cases: expected a list of at most 64 cases.') + benchmark['failure_cases'] = [_text(item, 'benchmark.failure_cases', 600) + for item in benchmark['failure_cases']] + if benchmark['status'] == 'blocked' and not benchmark['reason']: + raise CoverageError('A blocked benchmark needs an explicit reason.') + if benchmark['status'] == 'specified' and (not benchmark['environment'] + or not benchmark['success_predicate'] or not benchmark['failure_cases']): + raise CoverageError('A specified benchmark needs its environment, success predicate and failure cases.') + if type(row['evidence']) is not list or len(row['evidence']) > MAX_EVIDENCE: + raise CoverageError('evidence: expected a list of at most 1,200 declarations.') + evidence, evidence_ids = [], set() + for index, item in enumerate(row['evidence']): + path = f'evidence[{index}]' + item = _record(item, EVIDENCE_FIELDS, path) + evidence_id = _identity(item['id'], path + '.id') + if evidence_id in evidence_ids: + raise CoverageError('Evidence identities must be unique.') + evidence_ids.add(evidence_id) + digest = item['media_sha256'] + if type(digest) is not str or not re.fullmatch(r'[a-f0-9]{64}', digest): + raise CoverageError(path + '.media_sha256: expected a lowercase SHA-256 digest.') + if type(item['complete']) is not bool: + raise CoverageError(path + '.complete: expected an explicit boolean.') + entry = {'id': evidence_id, + 'plan_revision': _integer(item['plan_revision'], path + '.plan_revision', 1, 1_000_000), + 'case_id': _identity(item['case_id'], path + '.case_id'), + 'media_sha256': digest, 'complete': item['complete'], + 'outcome': _choice(item['outcome'], OUTCOMES, path + '.outcome'), + 'review_state': _choice(item['review_state'], REVIEW_STATES, path + '.review_state'), + 'review_origin': _choice(item['review_origin'], ('human', 'ai', 'unknown'), path + '.review_origin'), + 'rights_status': _choice(item['rights_status'], ('approved', 'unknown', 'denied'), path + '.rights_status'), + 'split': _choice(item['split'], ('train', 'dev', 'confirmation', 'unassigned'), path + '.split')} + for key in LINEAGE_FIELDS: + entry[key] = None if item[key] is None else _identity(item[key], path + '.' + key) + evidence.append(entry) + return {'format': FORMAT, 'schema_version': SCHEMA_VERSION, 'plan_id': identity, + 'plan_revision': revision, 'skill': skill, 'cases': cases, + 'benchmark': benchmark, 'evidence': evidence} + + +def _groups(evidence): + """Connect byte aliases and all declared lineage, including indirect links.""" + parent = list(range(len(evidence))) + + def find(index): + while parent[index] != index: + parent[index] = parent[parent[index]] + index = parent[index] + return index + + seen = {} + for index, item in enumerate(evidence): + for field in ('media_sha256',) + LINEAGE_FIELDS: + value = item[field] + if value is None: + continue + key = (field, value) + if key in seen: + left, right = find(index), find(seen[key]) + if left != right: + parent[left] = right + else: + seen[key] = index + return [find(index) for index in range(len(evidence))] + + +def summarize_plan(plan): + """Inspect declared planning coverage, without running media or benchmarks.""" + plan = validate_plan(plan) + cases = {item['id']: item for item in plan['cases']} + evidence = plan['evidence'] + groups = _groups(evidence) + group_splits, reviewed_outcomes = {}, {} + for group, item in zip(groups, evidence): + group_splits.setdefault(group, set()).add(item['split']) + if (item['plan_revision'] == plan['plan_revision'] and item['complete'] + and item['review_state'] == 'accepted' and item['review_origin'] == 'human' + and item['outcome'] != 'unknown'): + reviewed_outcomes.setdefault(item['media_sha256'], set()).add(item['outcome']) + by_case = {case_id: set() for case_id in cases} + accepted_groups, accepted_ids, exclusions = set(), [], [] + outcome_stats = {outcome: {'submitted': 0, 'accepted': 0} for outcome in OUTCOMES} + for group, item in zip(groups, evidence): + outcome_stats[item['outcome']]['submitted'] += 1 + reasons = [] + case = cases.get(item['case_id']) + if item['plan_revision'] != plan['plan_revision']: + reasons.append('stale plan revision') + if case is None: + reasons.append('case absent from this plan revision') + if not item['complete']: + reasons.append('partial attempt') + if item['review_state'] != 'accepted': + reasons.append('review is ' + item['review_state']) + if item['review_origin'] != 'human': + reasons.append('human review not declared') + if item['rights_status'] != 'approved': + reasons.append('intended-use rights are ' + item['rights_status']) + if any(item[key] is None for key in LINEAGE_FIELDS): + reasons.append('unknown recording, session or object lineage') + if item['outcome'] == 'unknown': + reasons.append('outcome unknown') + elif case is not None and item['outcome'] != case['outcome']: + reasons.append('outcome does not match the planned case') + if len(reviewed_outcomes.get(item['media_sha256'], ())) > 1: + reasons.append('identical media has conflicting accepted human outcome declarations') + splits = group_splits[group] + if 'confirmation' in splits: + reasons.append('connected group reserved for confirmation') + elif {'train', 'dev'} <= splits: + reasons.append('connected group crosses train and development splits') + elif item['split'] == 'unassigned': + reasons.append('split unassigned') + if reasons: + exclusions.append(item['id'] + ': ' + '; '.join(reasons) + '.') + continue + by_case[item['case_id']].add(group) + accepted_groups.add(group) + accepted_ids.append(item['id']) + outcome_stats[item['outcome']]['accepted'] += 1 + cells = [] + for case in plan['cases']: + count = len(by_case[case['id']]) + gap = max(0, case['required_groups'] - count) + cells.append({**case, 'accepted_groups': count, 'gap': gap, + 'status': 'met' if gap == 0 else 'gap'}) + gaps = sorted((cell for cell in cells if cell['gap']), key=lambda c: (-c['priority'], c['id'])) + next_capture = None + if gaps: + case = gaps[0] + why = (f"Planning target needs {case['gap']} more distinct declared source group(s). " + 'Recording alone does not meet the target; completeness, review and intended-use rights are required.') + if case['outcome'] == 'unknown': + why = 'Define this case\'s intended observed outcome before requesting qualifying evidence.' + next_capture = {'case_id': case['id'], 'label': case['label'], 'brief': case['brief'], 'why': why} + benchmark = plan['benchmark'] + warnings = [ + 'Planning targets are not scientifically sufficient dataset sizes or proof that a skillspace is exhausted.', + 'Counts use declared metadata only. This tool does not inspect videos or independently verify source novelty, reviews, outcomes or rights.', + 'Related sources stay connected through hashes, recordings, sessions and physical object instances; confirmation groups do not satisfy capture targets.', + 'A specified benchmark is a plan, not executed evidence. Coverage does not establish training readiness, acquired skill or hardware qualification.', + ] + if any(case['outcome'] == 'unknown' for case in plan['cases']): + warnings.append('Cases with an unknown intended outcome cannot satisfy a capture target.') + if plan['plan_id'].startswith('demo-'): + warnings.append('This example contains fabricated demonstration metadata, not actual footage or a published dataset.') + return {'format': 'dvidia.skillspace-coverage-summary', 'schema_version': 1, + 'plan_id': plan['plan_id'], 'plan_revision': plan['plan_revision'], + 'plan_sha256': hashlib.sha256(_canonical(plan)).hexdigest(), + 'skill': plan['skill'], + 'coverage': {'cells_met': sum(cell['status'] == 'met' for cell in cells), + 'cells_total': len(cells), 'accepted_groups': len(accepted_groups), + 'submitted_attempts': len(evidence), 'accepted_attempts': len(accepted_ids), + 'excluded_attempts': len(evidence) - len(accepted_ids), + 'all_cells_met': not gaps}, + 'cells': cells, 'gaps': gaps, 'next_capture': next_capture, + 'benchmark': {**benchmark, 'executed': False, 'qualified': False}, + 'outcomes': outcome_stats, 'warnings': warnings, 'exclusions': sorted(exclusions), + 'recognition': {'policy': 'Proposed recognition for reviewed, useful contributions; no automatic awards, points or cash.', + 'accepted_contributions': len(accepted_groups), + 'eligible_evidence_records': len(accepted_ids), + 'distinct_declared_groups': len(accepted_groups), + 'payments_active': False, 'awards_issued': False, + 'note': 'Recognition candidates are counted once per connected source group, not per alias or cut. These counts do not measure causal training benefit. Campaign terms and any attribution require a separate explicit agreement.'}, + 'robot_skill_acquired': False, 'physical_robot_ready': False} + + +def example_plan(): + """Fabricated planner example; no media, independent evidence or run implied.""" + cases = [ + {'id': 'nominal', 'label': 'Nominal placement', 'brief': 'Show one complete rigid-object placement, from initial state through release and stable final position.', 'outcome': 'success', 'required_groups': 2, 'priority': 6}, + {'id': 'new-instance', 'label': 'Different physical object', 'brief': 'Use another physical instance within the supported size and material envelope. Preserve its stable object identity.', 'outcome': 'success', 'required_groups': 2, 'priority': 8}, + {'id': 'empty-grasp', 'label': 'Empty grasp', 'brief': 'Record an observed empty closure with the jaws and object visible, preserving the failed outcome. Do not label commanded closure as holding.', 'outcome': 'failure', 'required_groups': 1, 'priority': 9}, + {'id': 'recovery', 'label': 'Recovery after missed grasp', 'brief': 'Capture the missed grasp, corrective attempt and final observed result in one complete episode.', 'outcome': 'recovery', 'required_groups': 2, 'priority': 10}, + {'id': 'occlusion', 'label': 'Brief visibility loss', 'brief': 'Capture a complete placement with a documented short visibility interruption and observable final state. Retain uncertainty where contact is not visible.', 'outcome': 'success', 'required_groups': 2, 'priority': 7}, + {'id': 'rotated-start', 'label': 'Rotated starting object', 'brief': 'Vary the initial object orientation within the supported envelope, keeping the whole attempt and final outcome visible.', 'outcome': 'success', 'required_groups': 2, 'priority': 5}, + ] + + def attempt(identity, case_id, digest, *, split='train', review='accepted', origin='human', rights='approved'): + return {'id': 'demo-' + identity, 'plan_revision': 1, 'case_id': case_id, + 'media_sha256': digest * 64, 'source_recording_id': 'demo-recording-' + identity, + 'session_id': 'demo-session-' + identity, 'object_instance_id': 'demo-object-' + identity, + 'outcome': next(case['outcome'] for case in cases if case['id'] == case_id), + 'complete': True, 'review_state': review, 'review_origin': origin, + 'rights_status': rights, 'split': split} + + return validate_plan({'format': FORMAT, 'schema_version': 1, 'plan_id': 'demo-rigid-placement', + 'plan_revision': 1, + 'skill': {'name': 'Rigid object placement · planning example', + 'goal': 'Move one supported rigid box onto a marked target and verify stable release.', + 'operating_envelope': 'Authored six-DOF arm with parallel jaws and bounded rigid-box table scene. This is a capture-planning example; shoes, deformables and physical hardware are outside the current adapter.'}, + 'cases': cases, + 'benchmark': {'status': 'specified', + 'environment': 'Proposed contact-pick-place-box-v0 simulation suite; no new suite is executed by this planner.', + 'reason': 'Descriptor only. Fresh frozen scenarios and artifact-bound results are still required.', + 'success_predicate': 'Observe grasp, lift, transfer, release and target stability using the independent native task gate; count every attempted trial.', + 'failure_cases': ['Empty grasp', 'Slip or drop', 'Unreachable start', 'Insufficient jaw force', 'Observation loss', 'Matched open-jaw negative control']}, + 'evidence': [attempt('nominal-a', 'nominal', 'a'), attempt('nominal-b', 'nominal', 'b'), + attempt('instance-a', 'new-instance', 'c', split='dev'), + attempt('failure-a', 'empty-grasp', 'd'), + attempt('recovery-proposal', 'recovery', 'e', review='proposed', origin='ai'), + attempt('occlusion-pending', 'occlusion', 'f', review='proposed', rights='unknown')]}) + + +def _output(value, path): + text = json.dumps(value, indent=2, ensure_ascii=False, allow_nan=False) + '\n' + if path is None: + print(text, end='') + else: + with Path(path).open('x', encoding='utf-8') as stream: + stream.write(text) + + +def main(argv=None): + parser = argparse.ArgumentParser(description='Inspect declared Skillspace capture coverage; no training, benchmark execution or rewards.') + commands = parser.add_subparsers(dest='command', required=True) + example = commands.add_parser('example', help='Emit fabricated demonstration metadata.') + example.add_argument('--output', type=Path, help='Create a new JSON file; existing files are not overwritten.') + inspect = commands.add_parser('inspect', help='Inspect one bounded coverage-plan JSON file.') + inspect.add_argument('plan', type=Path) + inspect.add_argument('--output', type=Path, help='Create a new summary JSON file.') + args = parser.parse_args(argv) + try: + if args.command == 'example': + value = example_plan() + else: + with args.plan.open('rb') as stream: + value = summarize_plan(stream.read(MAX_PLAN_BYTES + 1)) + _output(value, args.output) + except (CoverageError, OSError) as exc: + print('Coverage planning failed: ' + str(exc), file=sys.stderr) + return 1 + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/src/dvidia_training/coverage_studio.html b/src/dvidia_training/coverage_studio.html new file mode 100644 index 0000000..9976f3d --- /dev/null +++ b/src/dvidia_training/coverage_studio.html @@ -0,0 +1,53 @@ + + + + +Skillspace coverage · DVIDIA + + + + +
+
Local research prototype

Cover the skill, one useful example at a time.

Define the conditions your skill should handle, see where evidence is missing, and give contributors a specific next recording to make.

Loading an illustrative collection plan…

+

Connecting to the local planner…

+
Planned cases with enough reviewed groups
Not checked
Training readiness
Not evaluated
Simulation qualification
Not run
+

Collection targets are planning goals. Meeting them does not establish that a robot has learned the skill. This tool tallies supplied review and lineage metadata; it does not inspect videos or verify those declarations.

+

The next useful recording

Check actual footage
+

Coverage of the operating envelope

+

Every skill starts with a simulation benchmark

    This is a benchmark specification. Qualification requires an executable task environment, a frozen protocol and artifact-bound results for every attempted trial. A skill without a suitable environment stays blocked.

    +

    Recognize contributions that close a gap

    Make each recording request explain why it matters. Return a review receipt showing whether the attempt was accepted, needs correction, or adds a useful variation.

    • Recognize approved independent sessions and underrepresented conditions.
    • Recognize honest failures, recovery attempts and useful label corrections.
    • Show opt-in dataset credit and team coverage milestones.
    • For future funded campaigns, state the budget and acceptance terms before anyone records.

    Incentive design proposal. This prototype issues no rewards, points, payments or public contributor credit. More cuts from one source do not create independent evidence.

    +

    Define and revise the collection plan

    Saving a changed plan creates a new revision. Earlier evidence keeps its review history and needs review against the new plan before it can satisfy targets.

    +
    +
    Import or inspect a coverage-plan JSON

    This companion plan is separate from the footage manifest. Importing review metadata does not validate the underlying recording, permissions or reviewer judgment.

    +
    Declared evidence and exclusions

    AttemptCaseOutcomeReviewDataset use
      +
      + +
      + + + diff --git a/src/dvidia_training/episode_tape.py b/src/dvidia_training/episode_tape.py new file mode 100644 index 0000000..1b92ef4 --- /dev/null +++ b/src/dvidia_training/episode_tape.py @@ -0,0 +1,506 @@ +"""Versioned, data-only control-boundary tapes for the authored simulator. + +This is an evidence recorder, not a learner, hardware adapter or extension of +the existing movement sidecar. Validation checks the contract and consistency; +it does not establish that sensor readings or task labels are true. +""" +from __future__ import annotations + +import argparse +from copy import deepcopy +from dataclasses import asdict +from hashlib import sha256 +import json +import math +from pathlib import Path +import re + +KIND = 'dvidia.simulation-episode-tape' +ARM_PROFILE = 'dvidia-authored-6dof-parallel-jaw-v0' +TASK_ID = 'ArmPickPlace-v0' +MAX_BYTES = 64 * 1024 * 1024 +MAX_STEPS = 10000 +MAX_NATIVE_TICKS_PER_INTERVAL = 1000000 +JOINT_LIMITS = ((-2.6, 2.6), (-2.2, 1.6), (-2.7, 2.7), (-3., 3.), (-2.8, 2.8), (-3.1, 3.1)) +PHASES = {'observe', 'approach', 'descend', 'close', 'lift', 'transfer', 'place', + 'release', 'retreat', 'verify', 'recover_wait', 'recover_up', 'failed'} +CASES = ('nominal', 'open-jaw', 'release-after-lift') +SIGNALS = { + 'joint_position': {'shape': [6], 'units': 'rad', 'frame': 'joint_coordinates', 'privileged': True}, + 'joint_velocity': {'shape': [6], 'units': 'rad/s', 'frame': 'joint_coordinates', 'privileged': True}, + 'joint_actuator_torque': {'shape': [6], 'units': 'N*m', 'frame': 'joint_coordinates', 'privileged': True}, + 'jaw_actuator_force': {'shape': [2], 'units': 'N', 'frame': 'jaw_slide_coordinates_left_right', 'privileged': True}, + 'gripper_width': {'shape': [], 'units': 'm', 'frame': 'jaw_gap', 'privileged': True}, + 'pad_normal_forces': {'shape': [2], 'units': 'N', 'frame': 'contact_normal_left_right', 'privileged': True}, + 'grasp_contacts': {'shape': [2], 'units': 'count', 'frame': 'left_right_object_contact', 'privileged': True}, + 'table_contacts': {'shape': [], 'units': 'count', 'frame': 'table_object_contact', 'privileged': True}, + 'end_effector_position': {'shape': [3], 'units': 'm', 'frame': 'simulation_world', 'privileged': True}, + 'end_effector_quaternion': {'shape': [4], 'units': 'unit_quaternion_wxyz', 'frame': 'simulation_world', 'privileged': True}, + 'end_effector_velocity': {'shape': [3], 'units': 'm/s', 'frame': 'simulation_world', 'privileged': True}, + 'object_position': {'shape': [3], 'units': 'm', 'frame': 'simulation_world', 'privileged': True}, + 'object_quaternion': {'shape': [4], 'units': 'unit_quaternion_wxyz', 'frame': 'simulation_world', 'privileged': True}, + 'object_velocity': {'shape': [3], 'units': 'm/s', 'frame': 'simulation_world', 'privileged': True}, + 'object_angular_velocity': {'shape': [3], 'units': 'rad/s', 'frame': 'simulation_world', 'privileged': True}, + 'target_position': {'shape': [3], 'units': 'm', 'frame': 'simulation_world', 'privileged': True}, +} +PROTOCOL = { + 'revision': 'control-boundary-v1', + 'sampling': 'before decision and after each control step; intermediate native ticks are not retained', + 'application': 'native targets ramp during the step; applied values are the final native tick only', + 'sensor_age_limit_seconds': 0.1, + 'labels': 'authored controller action phase and next phase; evidence only, not deployment inputs', + 'qualification': 'simulation-only; privileged state; no physical robot qualification', +} +CONFIG_FIELDS = {'timestep', 'control_dt', 'horizon', 'settle_time', 'object_position', + 'target_position', 'object_half_size', 'object_mass', 'object_friction', + 'table_height', 'position_tolerance', 'dwell_seconds', 'scene_jitter', + 'joint_torque_limit', 'jaw_force_limit', 'joint_target_rate_limit', + 'jaw_width_rate_limit', 'gravity'} +SOURCE_FILES = {'arm_env.py', 'arm_policy.py', 'episode_tape.py'} + + +def canonical(document): + return json.dumps(document, sort_keys=True, separators=(',', ':'), allow_nan=False).encode('utf8') + + +def digest(document): + return sha256(canonical(document)).hexdigest() + + +def _keys(value, keys, label): + if type(value) is not dict or set(value) != set(keys): + raise ValueError(f'{label}: unknown or missing fields.') + + +def _number(value, label, *, minimum=None): + try: + finite = type(value) in (int, float) and math.isfinite(value) + except OverflowError: + finite = False + if not finite or (minimum is not None and value < minimum): + raise ValueError(f'{label}: finite numeric value required.') + return value + + +def _integer(value, label, *, minimum=0): + if type(value) is not int or value < minimum: + raise ValueError(f'{label}: integer >= {minimum} required.') + return value + + +def _ticks(numerator, denominator, label, *, maximum, tolerance=1e-9, boundary_roundoff=False): + """Check finite bounded ratios before rounding; extreme JSON is not a crash.""" + try: + ratio = numerator/denominator + except (OverflowError, ZeroDivisionError) as exc: + raise ValueError(f'{label}: invalid tick ratio.') from exc + lower = 1-tolerance if boundary_roundoff else 1 + if not math.isfinite(ratio) or not lower <= ratio <= maximum+tolerance: + raise ValueError(f'{label}: tick count must lie in [1, {maximum}].') + ticks = round(ratio) + if not 1 <= ticks <= maximum or not math.isclose(ratio, ticks, rel_tol=0, abs_tol=tolerance): + raise ValueError(f'{label}: an integer number of ticks is required.') + return ticks + + +def _identifier(value, label): + if type(value) is not str or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.:-]{0,159}', value): + raise ValueError(f'{label}: short source identifier required.') + + +def _sha(value, label): + if type(value) is not str or not re.fullmatch(r'[0-9a-f]{64}', value): + raise ValueError(f'{label}: lowercase SHA256 required.') + + +def _vector(value, size, label): + if type(value) is not list or len(value) != size: + raise ValueError(f'{label}: {size} entries required.') + for item in value: + _number(item, label) + + +def _decode(document): + if type(document) is dict: + try: + raw = canonical(document) + except (TypeError, ValueError, RecursionError, OverflowError) as exc: + raise ValueError('Tape must be finite JSON data.') from exc + elif isinstance(document, (str, bytes)): + raw = document.encode('utf8') if isinstance(document, str) else document + else: + raise ValueError('Tape must be a plain dict or UTF-8 JSON.') + if len(raw) > MAX_BYTES: + raise ValueError('Episode tape exceeds 64 MiB.') + def pairs(items): + row = {} + for key, value in items: + if key in row: + raise ValueError('Duplicate episode-tape field.') + row[key] = value + return row + def constant(value): + raise ValueError('Nonfinite episode-tape value.') + try: + return json.loads(raw, object_pairs_hook=pairs, parse_constant=constant) + except (UnicodeError, RecursionError, json.JSONDecodeError) as exc: + raise ValueError('Tape must be finite UTF-8 JSON.') from exc + + +def _snapshot(snapshot, label): + _keys(snapshot, {'timestamp_seconds', 'signals'}, label) + timestamp = _number(snapshot['timestamp_seconds'], label, minimum=0) + _keys(snapshot['signals'], SIGNALS, f'{label}.signals') + for name, contract in SIGNALS.items(): + sensor = snapshot['signals'][name] + _keys(sensor, {'status', 'captured_at_seconds', 'age_seconds', 'value'}, f'{label}.{name}') + if sensor['status'] == 'missing': + if any(sensor[key] is not None for key in ('captured_at_seconds', 'age_seconds', 'value')): + raise ValueError('Missing signals must have null time, age and value, never synthetic zeros.') + continue + if sensor['status'] not in ('available', 'stale'): + raise ValueError('Signal status must be available, stale or missing.') + captured = _number(sensor['captured_at_seconds'], name, minimum=0) + age = _number(sensor['age_seconds'], name, minimum=0) + if captured > timestamp + 1e-9 or not math.isclose(timestamp-captured, age, rel_tol=0, abs_tol=1e-9): + raise ValueError('Signal acquisition time and age disagree with observation time.') + if (sensor['status'] == 'available') != (age <= PROTOCOL['sensor_age_limit_seconds']): + raise ValueError('Signal freshness disagrees with its declared age.') + value = sensor['value'] + if contract['shape']: + _vector(value, contract['shape'][0], name) + else: + _number(value, name) + values = value if type(value) is list else [value] + if contract['units'] == 'count': + for item in values: + _integer(item, name) + if name in ('gripper_width', 'pad_normal_forces') and any(x < -1e-9 for x in values): + raise ValueError('Measured gap and normal forces cannot be negative.') + if contract['units'] == 'unit_quaternion_wxyz' and not .99 <= math.hypot(*value) <= 1.01: + raise ValueError('Measured orientation must be a unit quaternion.') + return timestamp + + +def _action(action, label): + _keys(action, {'joint_targets', 'gripper_width'}, label) + _vector(action['joint_targets'], 6, label) + for value, (low, high) in zip(action['joint_targets'], JOINT_LIMITS): + if not low <= value <= high: + raise ValueError('Requested joint targets exceed the authored arm limits.') + if not 0 <= _number(action['gripper_width'], label) <= .08: + raise ValueError('Requested gap exceeds the authored jaw limits.') + + +def validate_tape(document): + """Return a detached JSON tape; consistency and hashes are not qualification.""" + tape = _decode(document) + _keys(tape, {'schema_version', 'kind', 'scope', 'physical_robot_ready', 'task_id', + 'arm_profile', 'provenance', 'clock', 'signal_contract', 'protocol', + 'protocol_sha256', 'configuration', 'initial', 'steps', 'outcome'}, 'tape') + expected = {'schema_version': 1, 'kind': KIND, 'scope': 'simulation-only', + 'physical_robot_ready': False, 'task_id': TASK_ID, 'arm_profile': ARM_PROFILE} + if any(type(tape[key]) is not type(value) or tape[key] != value for key, value in expected.items()): + raise ValueError('Unsupported episode-tape version, arm, task or scope.') + if canonical(tape['signal_contract']) != canonical(SIGNALS) or canonical(tape['protocol']) != canonical(PROTOCOL) or tape['protocol_sha256'] != digest(PROTOCOL): + raise ValueError('Unsupported signal contract or recording protocol.') + provenance = tape['provenance'] + _keys(provenance, {'episode_id', 'source_recording_id', 'session_id', 'source_kind', + 'seed', 'case', 'policy_origin', 'policy_revision', 'engine', + 'scene_sha256', 'configuration_sha256', 'source_files_sha256'}, 'provenance') + for key in ('episode_id', 'source_recording_id', 'session_id', 'policy_revision'): + _identifier(provenance[key], key) + _integer(provenance['seed'], 'seed') + if provenance['source_kind'] != 'native-simulation-telemetry' or provenance['policy_origin'] != 'authored controller' or provenance['engine'] != 'mujoco-3.15.0' or provenance['case'] not in CASES: + raise ValueError('Unsupported simulation provenance.') + _sha(provenance['scene_sha256'], 'scene_sha256') + _sha(provenance['configuration_sha256'], 'configuration_sha256') + _keys(provenance['source_files_sha256'], SOURCE_FILES, 'source_files_sha256') + for name, value in provenance['source_files_sha256'].items(): + _sha(value, name) + config = tape['configuration'] + _keys(config, CONFIG_FIELDS, 'configuration') + for name, value in config.items(): + if name in ('object_position', 'target_position', 'object_half_size'): + _vector(value, 3, name) + else: + _number(value, name, minimum=0) + if config['timestep'] <= 0 or config['control_dt'] < config['timestep'] or config['horizon'] <= 0: + raise ValueError('Invalid simulator timing configuration.') + control_ticks = _ticks(config['control_dt'], config['timestep'], 'Control interval', maximum=MAX_NATIVE_TICKS_PER_INTERVAL) + _ticks(config['horizon'], config['control_dt'], 'Episode horizon', maximum=MAX_STEPS) + settle_ticks = _ticks(config['settle_time'], config['timestep'], 'Reset settling interval', maximum=MAX_NATIVE_TICKS_PER_INTERVAL) + if any(config[name] <= 0 for name in ('joint_torque_limit', 'jaw_force_limit', 'joint_target_rate_limit', 'jaw_width_rate_limit', 'gravity')): + raise ValueError('Native actuator and rate limits must be positive.') + for name, low, high in (('object_mass', .02, .08), ('object_friction', .4, 1.5), + ('position_tolerance', .01, .035), ('dwell_seconds', .15, .6), ('scene_jitter', 0., .01)): + if not low <= config[name] <= high: + raise ValueError('Configuration exceeds the authored simulation envelope.') + if config['table_height'] != .29 or any(not .015 <= x <= .025 for x in config['object_half_size']): + raise ValueError('Unsupported table height or rigid-box dimensions.') + for name in ('object_position', 'target_position'): + x, y, z = config[name] + if not .25 <= x <= .60 or not -.20 <= y <= .20 or math.hypot(x, y) > .56 or abs(z-(.29+config['object_half_size'][2])) > .01000001: + raise ValueError('Placement exceeds the authored table and reach envelope.') + if math.dist(config['object_position'][:2], config['target_position'][:2]) < .075: + raise ValueError('Placement must require nontrivial movement.') + if provenance['configuration_sha256'] != digest(config): + raise ValueError('Configuration hash mismatch.') + _keys(tape['clock'], {'kind', 'origin', 'native_time_offset_seconds'}, 'clock') + if tape['clock']['kind'] != 'simulation-elapsed-seconds' or tape['clock']['origin'] != 'after-reset-settle': + raise ValueError('Unsupported clock; wall time and device-clock mappings require a new contract.') + clock_offset = _number(tape['clock']['native_time_offset_seconds'], 'native clock offset', minimum=0) + expected_offset = settle_ticks*config['timestep'] + if not math.isfinite(expected_offset) or not math.isclose(clock_offset, expected_offset, rel_tol=0, abs_tol=1e-9): + raise ValueError('Native clock origin does not match the declared reset settling ticks.') + if _snapshot(tape['initial'], 'initial') != 0: + raise ValueError('Tape must begin at the declared zero-time origin.') + if type(tape['steps']) is not list or not 1 <= len(tape['steps']) <= MAX_STEPS: + raise ValueError('Tape requires 1–10000 control steps.') + previous = tape['initial'] + terminal = None + for index, step in enumerate(tape['steps']): + _keys(step, {'index', 'before', 'command', 'applied', 'after', 'labels', 'result'}, f'step {index}') + if type(step['index']) is not int or step['index'] != index or step['before'] != previous: + raise ValueError('Steps must be sequential and preserve the preceding observed state.') + start = _snapshot(step['before'], 'before') + end = _snapshot(step['after'], 'after') + if not start < end or end-start > config['control_dt']+1e-9: + raise ValueError('Control timestamps must advance without gaps or reordered samples.') + if end > config['horizon']+1e-9: + raise ValueError('Recorded transition exceeds the declared episode horizon.') + _ticks(end-start, config['timestep'], 'Recorded transition', maximum=control_ticks, tolerance=1e-7, boundary_roundoff=True) + command = step['command'] + _keys(command, {'issued_at_seconds', 'policy_requested', 'issued', 'intervention'}, 'command') + if _number(command['issued_at_seconds'], 'command issue time', minimum=0) != start: + raise ValueError('Command issue time must match its input observation.') + _action(command['policy_requested'], 'policy request') + _action(command['issued'], 'issued command') + if command['intervention'] not in ('none', 'open-jaw', 'release-after-lift'): + raise ValueError('Unknown recording intervention.') + if provenance['case'] == 'open-jaw' and command['intervention'] != 'open-jaw': + raise ValueError('Open-jaw case must declare every jaw-command override.') + if command['intervention'] == 'none' and command['issued'] != command['policy_requested']: + raise ValueError('Unmodified command must equal policy intent.') + if command['intervention'] != 'none': + if command['intervention'] != provenance['case'] or command['issued']['gripper_width'] != .08 or command['issued']['joint_targets'] != command['policy_requested']['joint_targets']: + raise ValueError('Jaw intervention must preserve authored joint intent and declare its case.') + applied = step['applied'] + _keys(applied, {'interval_start_seconds', 'interval_end_seconds', 'ramped_joint_targets', + 'ramped_gripper_width', 'native_position_controls', 'rate_limited'}, 'applied') + if _number(applied['interval_start_seconds'], 'application start', minimum=0) != start or _number(applied['interval_end_seconds'], 'application end', minimum=0) != end: + raise ValueError('Applied-command interval must match the recorded transition.') + _action({'joint_targets': applied['ramped_joint_targets'], 'gripper_width': applied['ramped_gripper_width']}, 'ramped command') + _vector(applied['native_position_controls'], 8, 'native_position_controls') + _action({'joint_targets': applied['native_position_controls'][:6], 'gripper_width': sum(applied['native_position_controls'][6:])}, 'native servo controls') + if any(not 0 <= x <= .04 for x in applied['native_position_controls'][6:]) or type(applied['rate_limited']) is not bool: + raise ValueError('Invalid jaw servo controls or rate-limit flag.') + expected_jaw_control = applied['ramped_gripper_width']/2 + if any(not math.isclose(x, expected_jaw_control, rel_tol=0, abs_tol=1e-12) for x in applied['native_position_controls'][6:]): + raise ValueError('Both native jaw controls must equal half the recorded ramped gap.') + labels = step['labels'] + _keys(labels, {'origin', 'action_phase', 'next_phase'}, 'labels') + if labels['origin'] != 'authored controller' or type(labels['action_phase']) is not str or type(labels['next_phase']) is not str or labels['action_phase'] not in PHASES or labels['next_phase'] not in PHASES: + raise ValueError('Unsupported authored phase evidence.') + result = step['result'] + _keys(result, {'terminated', 'truncated', 'success', 'valid', 'reason'}, 'result') + if any(type(result[key]) is not bool for key in ('terminated', 'truncated', 'success', 'valid')): + raise ValueError('Result flags must be booleans.') + reason = result['reason'] + if reason not in ('running', 'success', 'horizon', 'invalid_state', 'solver_warning', 'object_dropped'): + raise ValueError('Unsupported result reason.') + expected_flags = {'terminated': reason not in ('running', 'horizon'), 'truncated': reason == 'horizon', + 'success': reason == 'success', 'valid': reason not in ('invalid_state', 'solver_warning', 'object_dropped')} + if any(result[key] != value for key, value in expected_flags.items()): + raise ValueError('Terminal, success and validity flags contradict the result reason.') + if reason == 'horizon' and not math.isclose(end, config['horizon'], rel_tol=0, abs_tol=1e-8): + raise ValueError('Horizon termination must reach the declared episode horizon.') + terminal = result['terminated'] or result['truncated'] + if terminal != (index == len(tape['steps'])-1): + raise ValueError('Only the final transition must be terminal.') + if not terminal and not math.isclose(end-start, config['control_dt'], rel_tol=0, abs_tol=1e-9): + raise ValueError('Nonterminal transition must cover one control interval.') + previous = step['after'] + outcome = tape['outcome'] + _keys(outcome, {'result', 'simulation_seconds', 'control_steps', 'policy_failure_reason', 'recovery_count'}, 'outcome') + if canonical(outcome['result']) != canonical(tape['steps'][-1]['result']) or _number(outcome['simulation_seconds'], 'outcome time', minimum=0) != previous['timestamp_seconds'] or type(outcome['control_steps']) is not int or outcome['control_steps'] != len(tape['steps']): + raise ValueError('Final outcome does not match the terminal transition.') + _integer(outcome['recovery_count'], 'recovery_count') + failure = outcome['policy_failure_reason'] + if failure is not None and (type(failure) is not str or not failure or len(failure) > 160): + raise ValueError('Invalid authored policy failure reason.') + return tape + + +def snapshot(observation, timestamp_seconds): + """Convert one native observation; no unavailable sensor is filled with zero.""" + signals = {} + for name in SIGNALS: + value = observation.get(name) + if name in ('pad_normal_forces', 'grasp_contacts') and type(value) is dict: + value = [value.get('left'), value.get('right')] + missing = value is None or value == [] or (type(value) is list and any(x is None for x in value)) + signals[name] = {'status': 'missing' if missing else 'available', + 'captured_at_seconds': None if missing else timestamp_seconds, + 'age_seconds': None if missing else 0., 'value': None if missing else deepcopy(value)} + return {'timestamp_seconds': timestamp_seconds, 'signals': signals} + + +def record_episode(*, episode_id, source_recording_id, session_id, seed=0, case='nominal', config=None): + """Record a fresh authored simulation. Requires the optional arm dependencies. + + No renderer, network, video conversion, training or frozen benchmark is run. + Case interventions change issued jaw commands, never object dynamics. + """ + for key, value in (('episode_id', episode_id), ('source_recording_id', source_recording_id), ('session_id', session_id)): + _identifier(value, key) + _integer(seed, 'seed') + if case not in CASES: + raise ValueError(f'Use one of: {", ".join(CASES)}.') + from .arm_env import ArmConfig, ArmEnv + from .arm_policy import ArmPickPlacePolicy, POLICY_REVISION + environment = ArmEnv(config if config is not None else ArmConfig()) + observation, info = environment.reset(seed=seed) + policy = ArmPickPlacePolicy(environment) + configuration = json.loads(canonical(asdict(environment.config))) + tape = {'schema_version': 1, 'kind': KIND, 'scope': 'simulation-only', 'physical_robot_ready': False, + 'task_id': TASK_ID, 'arm_profile': ARM_PROFILE, + 'provenance': {'episode_id': episode_id, 'source_recording_id': source_recording_id, + 'session_id': session_id, 'source_kind': 'native-simulation-telemetry', + 'seed': seed, 'case': case, 'policy_origin': 'authored controller', + 'policy_revision': POLICY_REVISION, 'engine': f'mujoco-{info["engine"]}', + 'scene_sha256': info['scene_sha256'], 'configuration_sha256': digest(configuration), + 'source_files_sha256': {name: sha256(Path(__file__).with_name(name).read_bytes()).hexdigest() for name in SOURCE_FILES}}, + 'clock': {'kind': 'simulation-elapsed-seconds', 'origin': 'after-reset-settle', + 'native_time_offset_seconds': float(environment.data.time)}, + 'signal_contract': deepcopy(SIGNALS), 'protocol': deepcopy(PROTOCOL), + 'protocol_sha256': digest(PROTOCOL), 'configuration': configuration, + 'initial': snapshot(observation, 0.), 'steps': []} + before = tape['initial'] + release_at = None + while True: + policy_requested = policy(observation) + issued = deepcopy(policy_requested) + diagnostics = policy.diagnostics() + timestamp = before['timestamp_seconds'] + if case == 'release-after-lift' and release_at is None and info['lift_seen']: + release_at = timestamp + intervention = 'none' + if case == 'open-jaw' or (release_at is not None and timestamp < release_at+.35): + issued['gripper_width'] = .08 + intervention = case + observation, _, terminated, truncated, info = environment.step(issued) + end = float(info['simulation_time']) + after = snapshot(observation, end) + if info['reason'] == 'invalid_state': + for sensor in after['signals'].values(): + sensor.update(status='missing', captured_at_seconds=None, age_seconds=None, value=None) + result = {'terminated': bool(terminated), 'truncated': bool(truncated), 'success': info['success'], + 'valid': info['valid'], 'reason': info['reason']} + tape['steps'].append({'index': len(tape['steps']), 'before': before, + 'command': {'issued_at_seconds': timestamp, 'policy_requested': deepcopy(policy_requested), + 'issued': issued, 'intervention': intervention}, + 'applied': {'interval_start_seconds': timestamp, 'interval_end_seconds': end, + 'ramped_joint_targets': info['commanded_joint_targets'], + 'ramped_gripper_width': info['commanded_gripper_width'], + 'native_position_controls': info['native_position_controls'], + 'rate_limited': info['command_rate_limited']}, + 'after': after, 'labels': {'origin': 'authored controller', + 'action_phase': diagnostics['action_phase'], 'next_phase': diagnostics['phase']}, + 'result': result}) + before = after + if terminated or truncated: + break + if len(tape['steps']) >= MAX_STEPS: + raise ValueError('Recording exceeds the episode-tape step limit.') + diagnostics = policy.diagnostics() + tape['outcome'] = {'result': result, 'simulation_seconds': end, 'control_steps': len(tape['steps']), + 'policy_failure_reason': diagnostics['failure_reason'], 'recovery_count': diagnostics['recovery_count']} + return validate_tape(tape) + + +def load_tape(path): + path = Path(path) + if path.stat().st_size > MAX_BYTES: + raise ValueError('Episode tape exceeds 64 MiB.') + return validate_tape(path.read_bytes()) + + +def write_tape(path, document): + """Create a new tape without overwriting an existing file or benchmark.""" + tape = validate_tape(document) + raw = canonical(tape) + if len(raw)+1 > MAX_BYTES: + raise ValueError('Episode tape exceeds 64 MiB.') + path = Path(path) + with path.open('xb') as output: + output.write(raw + b'\n') + return {'path': str(path), 'sha256': sha256(raw + b'\n').hexdigest(), 'bytes': len(raw)+1} + + +def inspect_tape(document): + """Summarize evidence and missingness without treating authored labels as truth.""" + tape = validate_tape(document) + counts = {name: {'available': 0, 'stale': 0, 'missing': 0} for name in SIGNALS} + phases = {} + max_age = {name: None for name in SIGNALS} + interventions = 0 + observations = [tape['initial']] + [step['after'] for step in tape['steps']] + for observation in observations: + for name, sensor in observation['signals'].items(): + counts[name][sensor['status']] += 1 + if sensor['age_seconds'] is not None: + max_age[name] = max(max_age[name] or 0., sensor['age_seconds']) + for step in tape['steps']: + phase = step['labels']['action_phase'] + phases[phase] = phases.get(phase, 0)+1 + interventions += step['command']['intervention'] != 'none' + return {'kind': 'dvidia.episode-tape-inspection', 'schema_version': 1, + 'scope': 'simulation-only', 'physical_robot_ready': False, + 'episode_id': tape['provenance']['episode_id'], 'source_recording_id': tape['provenance']['source_recording_id'], + 'session_id': tape['provenance']['session_id'], 'case': tape['provenance']['case'], + 'tape_content_sha256': digest(tape), 'protocol_sha256': tape['protocol_sha256'], + 'control_steps': len(tape['steps']), 'unique_observations': len(observations), + 'simulation_seconds': tape['outcome']['simulation_seconds'], 'outcome': deepcopy(tape['outcome']), + 'authored_action_phase_samples': phases, 'command_intervention_samples': interventions, + 'signal_status_counts': counts, 'max_signal_age_seconds': max_age, + 'limitation': 'Contract validation only; privileged simulator state and authored phase labels; no learned supervisor or hardware qualification.'} + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest='operation', required=True) + recorder = sub.add_parser('record', help='Record a fresh native simulation; requires the arm extra.') + recorder.add_argument('--output', required=True) + recorder.add_argument('--episode-id', required=True) + recorder.add_argument('--source-recording-id', required=True) + recorder.add_argument('--session-id', required=True) + recorder.add_argument('--seed', type=int, default=0) + recorder.add_argument('--case', choices=CASES, default='nominal') + for name in ('validate', 'inspect'): + reader = sub.add_parser(name, help='Read data-only tape without MuJoCo or a GPU.') + reader.add_argument('tape') + args = parser.parse_args(argv) + try: + if args.operation == 'record': + path = Path(args.output) + if path.exists(): + raise ValueError('Output already exists; use a fresh path.') + tape = record_episode(episode_id=args.episode_id, source_recording_id=args.source_recording_id, + session_id=args.session_id, seed=args.seed, case=args.case) + result = write_tape(path, tape) + result['inspection'] = inspect_tape(tape) + else: + tape = load_tape(args.tape) + result = inspect_tape(tape) if args.operation == 'inspect' else { + 'status': 'valid', 'scope': 'simulation-only', 'physical_robot_ready': False, + 'tape_content_sha256': digest(tape), 'control_steps': len(tape['steps'])} + print(json.dumps(result, sort_keys=True, allow_nan=False)) + return 0 + except (OSError, ValueError, ImportError, RuntimeError) as exc: + print(json.dumps({'status': 'error', 'error': str(exc)}, allow_nan=False)) + return 1 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/src/dvidia_training/training_studio.html b/src/dvidia_training/training_studio.html index 950016c..82194bd 100644 --- a/src/dvidia_training/training_studio.html +++ b/src/dvidia_training/training_studio.html @@ -10,7 +10,7 @@ diff --git a/src/dvidia_training/training_studio.py b/src/dvidia_training/training_studio.py index 88054e3..50721b3 100644 --- a/src/dvidia_training/training_studio.py +++ b/src/dvidia_training/training_studio.py @@ -336,6 +336,11 @@ def do_GET(self): local = True if self.path == '/': self.respond(Path(__file__).with_name('training_studio.html').read_bytes(), mime='text/html; charset=utf-8') + elif self.path == '/coverage': + self.respond(Path(__file__).with_name('coverage_studio.html').read_bytes(), mime='text/html; charset=utf-8') + elif self.path == '/api/coverage/example': + from .coverage import example_plan + self.respond(example_plan()) elif self.path == '/api/state': self.respond(self.server.state()) elif self.path == '/api/result': @@ -364,6 +369,10 @@ def do_POST(self): source, online, seed = self.train_request() kind = 'prepare' if self.path == '/api/prepare' else 'train' self.respond(self.server.submit(kind, source, online, seed), 202) + elif self.path == '/api/coverage/inspect': + from .coverage import MAX_PLAN_BYTES, summarize_plan, validate_plan + plan = validate_plan(self.body('application/json', MAX_PLAN_BYTES)) + self.respond({**summarize_plan(plan), 'plan': plan}) elif self.path == '/api/upload': raw = self.body('application/zip', UPLOAD_LIMIT) validate_zip(raw) diff --git a/tests/test_coverage.py b/tests/test_coverage.py new file mode 100644 index 0000000..3b95e11 --- /dev/null +++ b/tests/test_coverage.py @@ -0,0 +1,235 @@ +"""Capture declarations cannot fabricate independence, training or qualification.""" +from __future__ import annotations + +import contextlib +import copy +import io +import json +from pathlib import Path +import tempfile +import unittest + +from dvidia_training.coverage import (CoverageError, MAX_CASES, MAX_EVIDENCE, + MAX_PLAN_BYTES, example_plan, main, + summarize_plan, validate_plan) + + +def plan_with(*evidence): + plan = example_plan() + plan['cases'] = [plan['cases'][0]] + plan['cases'][0]['required_groups'] = 3 + plan['evidence'] = list(evidence) + return plan + + +def attempt(identity, **changes): + row = copy.deepcopy(example_plan()['evidence'][0]) + row.update(id=identity, media_sha256=f'{int(identity[-1]):064x}', + source_recording_id='recording-' + identity, + session_id='session-' + identity, object_instance_id='object-' + identity) + row.update(changes) + return row + + +class CoverageTests(unittest.TestCase): + def test_example_is_fabricated_and_summarizes_planning_only(self): + summary = summarize_plan(example_plan()) + self.assertEqual(summary['coverage'], {'cells_met': 2, 'cells_total': 6, + 'accepted_groups': 4, 'submitted_attempts': 6, + 'accepted_attempts': 4, 'excluded_attempts': 2, + 'all_cells_met': False}) + self.assertEqual(summary['next_capture']['case_id'], 'recovery') + self.assertFalse(summary['robot_skill_acquired']) + self.assertFalse(summary['physical_robot_ready']) + self.assertFalse(summary['benchmark']['executed']) + self.assertFalse(summary['benchmark']['qualified']) + self.assertFalse(summary['recognition']['payments_active']) + self.assertFalse(summary['recognition']['awards_issued']) + self.assertTrue(any('fabricated' in message for message in summary['warnings'])) + self.assertTrue(any('not scientifically sufficient' in message for message in summary['warnings'])) + self.assertEqual(summary['outcomes']['recovery'], {'submitted': 1, 'accepted': 0}) + + def test_aliases_and_cuts_do_not_manufacture_distinct_groups(self): + rows = [attempt('e1'), attempt('e2'), attempt('e3')] + rows[1]['media_sha256'] = rows[0]['media_sha256'] + rows[2]['source_recording_id'] = rows[1]['source_recording_id'] + summary = summarize_plan(plan_with(*rows)) + self.assertEqual(summary['coverage']['accepted_groups'], 1) + self.assertEqual(summary['cells'][0]['accepted_groups'], 1) + self.assertEqual(summary['cells'][0]['gap'], 2) + self.assertFalse(summary['coverage']['all_cells_met']) + self.assertEqual(summary['recognition']['accepted_contributions'], 1) + self.assertEqual(summary['recognition']['eligible_evidence_records'], 3) + + def test_conflicting_reviewed_outcomes_for_identical_media_cannot_be_cherry_picked(self): + first, second = attempt('e1'), attempt('e2', outcome='failure') + second['media_sha256'] = first['media_sha256'] + summary = summarize_plan(plan_with(first, second)) + self.assertEqual(summary['coverage']['accepted_groups'], 0) + self.assertTrue(all('conflicting accepted human outcome' in text for text in summary['exclusions'])) + second['review_state'] = 'proposed' + second['review_origin'] = 'ai' + summary = summarize_plan(plan_with(first, second)) + self.assertEqual(summary['coverage']['accepted_groups'], 1) + + def test_transitive_recording_session_object_connections_are_one_group(self): + rows = [attempt('e1'), attempt('e2'), attempt('e3'), attempt('e4')] + rows[1]['source_recording_id'] = rows[0]['source_recording_id'] + rows[2]['session_id'] = rows[1]['session_id'] + rows[3]['object_instance_id'] = rows[2]['object_instance_id'] + summary = summarize_plan(plan_with(*rows)) + self.assertEqual(summary['coverage']['accepted_groups'], 1) + self.assertEqual(summary['cells'][0]['accepted_groups'], 1) + + def test_unchanged_text_across_different_identity_names_does_not_connect_fields(self): + first, second = attempt('e1'), attempt('e2') + first['session_id'] = second['object_instance_id'] + summary = summarize_plan(plan_with(first, second)) + self.assertEqual(summary['coverage']['accepted_groups'], 2) + + def test_unknown_lineage_never_satisfies_a_quota(self): + for field in ('source_recording_id', 'session_id', 'object_instance_id'): + with self.subTest(field=field): + summary = summarize_plan(plan_with(attempt('e1', **{field: None}))) + self.assertEqual(summary['coverage']['accepted_groups'], 0) + self.assertEqual(summary['coverage']['excluded_attempts'], 1) + self.assertIn('unknown recording, session or object lineage', summary['exclusions'][0]) + + def test_partial_proposed_ai_rejected_rights_and_stale_evidence_excluded(self): + variations = [dict(complete=False), dict(review_state='proposed'), dict(review_origin='ai'), + dict(review_origin='unknown'), dict(review_state='rejected'), + dict(review_state='needs_changes'), dict(review_state='stale'), + dict(rights_status='unknown'), dict(rights_status='denied'), + dict(plan_revision=2), dict(split='unassigned'), dict(outcome='unknown'), + dict(outcome='failure'), dict(case_id='old-case')] + for changes in variations: + with self.subTest(changes=changes): + summary = summarize_plan(plan_with(attempt('e1', **changes))) + self.assertEqual(summary['coverage']['accepted_groups'], 0) + self.assertEqual(summary['coverage']['excluded_attempts'], 1) + self.assertIsNotNone(summary['next_capture']) + + def test_confirmation_reservation_excludes_entire_connected_group(self): + first, reserved, bridge = attempt('e1'), attempt('e2', split='confirmation'), attempt('e3') + bridge['session_id'] = first['session_id'] + bridge['object_instance_id'] = reserved['object_instance_id'] + summary = summarize_plan(plan_with(first, reserved, bridge)) + self.assertEqual(summary['coverage']['accepted_groups'], 0) + self.assertEqual(summary['coverage']['excluded_attempts'], 3) + self.assertTrue(all('reserved for confirmation' in text for text in summary['exclusions'])) + + def test_train_development_conflict_excludes_group_even_if_labels_accepted(self): + first, second = attempt('e1'), attempt('e2', split='dev') + second['session_id'] = first['session_id'] + summary = summarize_plan(plan_with(first, second)) + self.assertEqual(summary['coverage']['accepted_groups'], 0) + self.assertTrue(all('crosses train and development' in text for text in summary['exclusions'])) + + def test_pending_or_stale_alias_cannot_hide_a_confirmation_reservation(self): + first, holdout = attempt('e1'), attempt('e2', split='confirmation', review_state='proposed', plan_revision=7) + holdout['media_sha256'] = first['media_sha256'] + summary = summarize_plan(plan_with(first, holdout)) + self.assertEqual(summary['coverage']['accepted_groups'], 0) + + def test_met_targets_do_not_claim_data_sufficiency_or_robot_qualification(self): + plan = plan_with(attempt('e1'), attempt('e2'), attempt('e3')) + summary = summarize_plan(plan) + self.assertTrue(summary['coverage']['all_cells_met']) + self.assertEqual(summary['cells'][0]['status'], 'met') + self.assertIsNone(summary['next_capture']) + self.assertFalse(summary['robot_skill_acquired']) + self.assertFalse(summary['benchmark']['qualified']) + self.assertTrue(any('does not establish training readiness' in text for text in summary['warnings'])) + + def test_priority_ties_are_stable_and_unknown_cases_require_definition(self): + plan = example_plan() + plan['evidence'] = [] + plan['cases'][0].update(priority=10, id='aaa', outcome='unknown') + summary = summarize_plan(plan) + self.assertEqual(summary['next_capture']['case_id'], 'aaa') + self.assertIn('Define this case', summary['next_capture']['why']) + self.assertTrue(any('unknown intended outcome' in text for text in summary['warnings'])) + + def test_benchmark_is_mandatory_and_passed_or_qualified_cannot_be_declared(self): + mutations = [lambda plan: plan.pop('benchmark'), + lambda plan: plan['benchmark'].update(status='passed'), + lambda plan: plan['benchmark'].update(executed=True), + lambda plan: plan['benchmark'].update(qualified=True), + lambda plan: plan.update(robot_skill_acquired=True), + lambda plan: plan['benchmark'].update(environment=''), + lambda plan: plan['benchmark'].update(success_predicate=''), + lambda plan: plan['benchmark'].update(failure_cases=[])] + for mutate in mutations: + with self.subTest(mutate=mutate): + plan = example_plan() + mutate(plan) + with self.assertRaises(CoverageError): + validate_plan(plan) + plan = example_plan() + plan['benchmark'].update(status='blocked', environment='', reason='No compatible shoe adapter exists.', + success_predicate='', failure_cases=[]) + summary = summarize_plan(plan) + self.assertEqual(summary['benchmark']['status'], 'blocked') + self.assertIn('shoe adapter', summary['benchmark']['reason']) + plan['benchmark']['reason'] = '' + with self.assertRaises(CoverageError): + validate_plan(plan) + + def test_strict_fields_types_bounds_and_unique_ids(self): + mutations = [lambda plan: plan.update(schema_version=True), + lambda plan: plan.update(plan_revision=True), + lambda plan: plan['cases'][0].update(required_groups=True), + lambda plan: plan['cases'][0].update(priority=11), + lambda plan: plan['cases'][0].update(brief=''), + lambda plan: plan['skill'].update(goal='\x00'), + lambda plan: plan['evidence'][0].update(complete=1), + lambda plan: plan['evidence'][0].update(media_sha256='A'*64), + lambda plan: plan['evidence'][0].update(session_id=''), + lambda plan: plan['evidence'][0].update(arbitrary_path='/tmp/file'), + lambda plan: plan['cases'].append(copy.deepcopy(plan['cases'][0])), + lambda plan: plan['evidence'].append(copy.deepcopy(plan['evidence'][0])), + lambda plan: plan.update(cases=[copy.deepcopy(plan['cases'][0])] * (MAX_CASES + 1)), + lambda plan: plan.update(evidence=[copy.deepcopy(plan['evidence'][0])] * (MAX_EVIDENCE + 1))] + for mutate in mutations: + with self.subTest(mutate=mutate): + plan = example_plan() + mutate(plan) + with self.assertRaises(CoverageError): + validate_plan(plan) + + def test_json_rejects_duplicates_nonfinite_unicode_and_oversized_input(self): + for raw in ('{"format":1,"format":2}', '{"x":NaN}', '{"x":1e999}', b'\xff', + ' ' * (MAX_PLAN_BYTES + 1), {'x': float('nan')}, {'x': '\ud800'}): + with self.subTest(raw=str(raw)[:40]): + with self.assertRaises(CoverageError): + validate_plan(raw) + + def test_normalization_is_detached_and_summary_hash_is_deterministic(self): + original = example_plan() + normalized = validate_plan(original) + normalized['benchmark']['failure_cases'].append('Changed elsewhere') + self.assertNotIn('Changed elsewhere', original['benchmark']['failure_cases']) + raw = json.dumps(original, sort_keys=True).encode() + self.assertEqual(summarize_plan(original), summarize_plan(raw)) + + def test_cli_example_inspect_bounded_read_and_no_overwrite(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / 'plan.json' + summary_path = Path(directory) / 'summary.json' + self.assertEqual(main(['example', '--output', str(path)]), 0) + self.assertEqual(main(['inspect', str(path), '--output', str(summary_path)]), 0) + summary = json.loads(summary_path.read_text()) + self.assertFalse(summary['benchmark']['executed']) + original = path.read_bytes() + error = io.StringIO() + with contextlib.redirect_stderr(error): + self.assertEqual(main(['example', '--output', str(path)]), 1) + self.assertEqual(path.read_bytes(), original) + self.assertIn('Coverage planning failed', error.getvalue()) + path.write_bytes(b' ' * (MAX_PLAN_BYTES + 1)) + with contextlib.redirect_stderr(io.StringIO()): + self.assertEqual(main(['inspect', str(path)]), 1) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_episode_tape.py b/tests/test_episode_tape.py new file mode 100644 index 0000000..b169666 --- /dev/null +++ b/tests/test_episode_tape.py @@ -0,0 +1,357 @@ +"""Evidence contract, time alignment and native recording checks.""" +from copy import deepcopy +from contextlib import redirect_stdout +from importlib.util import find_spec +import io +import json +from pathlib import Path +import subprocess +import sys +from tempfile import TemporaryDirectory +import unittest + +from dvidia_training.episode_tape import (ARM_PROFILE, KIND, PROTOCOL, SIGNALS, TASK_ID, + canonical, digest, inspect_tape, load_tape, + main, record_episode, snapshot, validate_tape, write_tape) + + +def fixture(): + observation = {name: ([0.] * contract['shape'][0] if contract['shape'] else 0.) + for name, contract in SIGNALS.items()} + observation.update(end_effector_quaternion=[1., 0., 0., 0.], object_quaternion=[1., 0., 0., 0.], + grasp_contacts={'left': 0, 'right': 0}, table_contacts=1, + pad_normal_forces={'left': 0., 'right': 0.}, gripper_width=.08) + initial = snapshot(observation, 0.) + after = snapshot(observation, .02) + action = {'joint_targets': [0.] * 6, 'gripper_width': .08} + config = {'timestep': .001, 'control_dt': .02, 'horizon': .02, 'settle_time': .4, + 'object_position': [.42, -.04, .31], 'target_position': [.54, .10, .31], + 'object_half_size': [.02, .02, .02], 'object_mass': .04, 'object_friction': .8, + 'table_height': .29, 'position_tolerance': .025, 'dwell_seconds': .25, + 'scene_jitter': 0., 'joint_torque_limit': 40., 'jaw_force_limit': 15., + 'joint_target_rate_limit': 3., 'jaw_width_rate_limit': .16, 'gravity': 9.81} + result = {'terminated': False, 'truncated': True, 'success': False, 'valid': True, 'reason': 'horizon'} + return {'schema_version': 1, 'kind': KIND, 'scope': 'simulation-only', 'physical_robot_ready': False, + 'task_id': TASK_ID, 'arm_profile': ARM_PROFILE, + 'provenance': {'episode_id': 'fixture-00', 'source_recording_id': 'source-00', 'session_id': 'session-00', + 'source_kind': 'native-simulation-telemetry', 'seed': 0, 'case': 'nominal', + 'policy_origin': 'authored controller', 'policy_revision': 'test-fixture', + 'engine': 'mujoco-3.15.0', 'scene_sha256': '0'*64, 'configuration_sha256': digest(config), + 'source_files_sha256': {name: '0'*64 for name in ('arm_env.py', 'arm_policy.py', 'episode_tape.py')}}, + 'clock': {'kind': 'simulation-elapsed-seconds', 'origin': 'after-reset-settle', 'native_time_offset_seconds': .4}, + 'signal_contract': deepcopy(SIGNALS), 'protocol': deepcopy(PROTOCOL), 'protocol_sha256': digest(PROTOCOL), + 'configuration': config, 'initial': initial, + 'steps': [{'index': 0, 'before': deepcopy(initial), + 'command': {'issued_at_seconds': 0., 'policy_requested': deepcopy(action), 'issued': deepcopy(action), 'intervention': 'none'}, + 'applied': {'interval_start_seconds': 0., 'interval_end_seconds': .02, + 'ramped_joint_targets': [0.] * 6, 'ramped_gripper_width': .08, + 'native_position_controls': [0.] * 6+[.04, .04], 'rate_limited': False}, + 'after': after, 'labels': {'origin': 'authored controller', 'action_phase': 'approach', 'next_phase': 'approach'}, + 'result': result}], + 'outcome': {'result': deepcopy(result), 'simulation_seconds': .02, 'control_steps': 1, + 'policy_failure_reason': None, 'recovery_count': 0}} + + +class EpisodeTapeContractTests(unittest.TestCase): + def test_validate_detaches_input_and_inspection_counts_unique_observations(self): + tape = fixture() + validated = validate_tape(tape) + validated['initial']['signals']['joint_position']['value'][0] = 1. + self.assertEqual(tape['initial']['signals']['joint_position']['value'][0], 0.) + summary = inspect_tape(tape) + self.assertEqual(summary['unique_observations'], 2) + self.assertEqual(summary['signal_status_counts']['joint_position'], {'available': 2, 'stale': 0, 'missing': 0}) + self.assertFalse(summary['physical_robot_ready']) + + def test_read_only_operations_do_not_import_optional_native_or_numpy_packages(self): + code = '''import builtins +original = builtins.__import__ +def guarded(name, *args, **kwargs): + if name.split('.')[0] in ('mujoco', 'numpy'): + raise RuntimeError('Optional dependency imported') + return original(name, *args, **kwargs) +builtins.__import__ = guarded +from dvidia_training.episode_tape import validate_tape, inspect_tape +from tests.test_episode_tape import fixture +assert inspect_tape(validate_tape(fixture()))['control_steps'] == 1 +''' + result = subprocess.run([sys.executable, '-c', code], cwd=Path(__file__).resolve().parents[1], capture_output=True, text=True) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_sensor_missingness_and_staleness_are_explicit_and_counted(self): + tape = fixture() + missing = {'status': 'missing', 'captured_at_seconds': None, 'age_seconds': None, 'value': None} + for snapshot_ in (tape['initial'], tape['steps'][0]['before'], tape['steps'][0]['after']): + snapshot_['signals']['pad_normal_forces'] = deepcopy(missing) + # An observation may be stale without changing time order. It is not replaced with zeros. + tape['configuration']['control_dt'] = .2 + tape['configuration']['horizon'] = .2 + tape['provenance']['configuration_sha256'] = digest(tape['configuration']) + tape['steps'][0]['after']['timestamp_seconds'] = .2 + for sensor in tape['steps'][0]['after']['signals'].values(): + if sensor['status'] != 'missing': + sensor.update(captured_at_seconds=.2) + tape['steps'][0]['after']['signals']['joint_velocity'].update(status='stale', captured_at_seconds=0., age_seconds=.2) + tape['steps'][0]['applied']['interval_end_seconds'] = .2 + tape['outcome']['simulation_seconds'] = .2 + summary = inspect_tape(tape) + self.assertEqual(summary['signal_status_counts']['pad_normal_forces']['missing'], 2) + self.assertEqual(summary['signal_status_counts']['joint_velocity']['stale'], 1) + self.assertEqual(summary['max_signal_age_seconds']['joint_velocity'], .2) + + def test_absent_observation_field_is_not_a_zero_measurement(self): + partial = snapshot({}, 0.) + for sensor in partial['signals'].values(): + self.assertEqual(sensor, {'status': 'missing', 'captured_at_seconds': None, 'age_seconds': None, 'value': None}) + + def test_duplicate_fields_and_nonfinite_json_are_refused(self): + for raw in ('{"kind": "a", "kind": "b"}', '{"value": NaN}', '{"value": Infinity}', b'\xff'): + with self.subTest(raw=raw), self.assertRaises(ValueError): + validate_tape(raw) + + def test_unknown_keys_are_refused_at_nested_contract_boundaries(self): + paths = [(), ('provenance',), ('clock',), ('initial',), ('initial', 'signals', 'joint_position'), + ('steps', 0), ('steps', 0, 'command'), ('steps', 0, 'applied'), ('steps', 0, 'labels'), + ('steps', 0, 'result'), ('outcome',)] + for path in paths: + tape = fixture() + target = tape + for key in path: + target = target[key] + target['unexpected'] = 'unchecked' + with self.subTest(path=path), self.assertRaises(ValueError): + validate_tape(tape) + + def test_nonfinite_boolean_shape_and_quaternion_measurements_are_refused(self): + corruptions = [('joint_position', [0.]*5), ('joint_position', [True]+[0.]*5), + ('joint_position', [float('nan')]+[0.]*5), ('joint_position', [10**1000]+[0.]*5), + ('object_quaternion', [0.]*4), ('grasp_contacts', [0., 1.]), + ('gripper_width', -.001), ('pad_normal_forces', [-1., 0.])] + for name, value in corruptions: + tape = fixture() + tape['steps'][0]['after']['signals'][name]['value'] = value + with self.subTest(name=name, value=value), self.assertRaises(ValueError): + validate_tape(tape) + + def test_sensor_age_and_missingness_contradictions_are_refused(self): + for update in ({'age_seconds': .001}, {'captured_at_seconds': .03}, {'status': 'stale'}, + {'status': 'missing'}, {'status': 'unknown'}): + tape = fixture() + tape['steps'][0]['after']['signals']['joint_position'].update(update) + with self.subTest(update=update), self.assertRaises(ValueError): + validate_tape(tape) + + def test_time_reordering_input_discontinuity_and_application_mismatch_are_refused(self): + for target, field, value in [('after', 'timestamp_seconds', 0.), ('applied', 'interval_start_seconds', .001), + ('applied', 'interval_end_seconds', .01), ('command', 'issued_at_seconds', .001), + ('command', 'issued_at_seconds', False)]: + tape = fixture() + tape['steps'][0][target][field] = value + with self.subTest(target=target, field=field), self.assertRaises(ValueError): + validate_tape(tape) + tape = fixture() + tape['steps'][0]['before']['signals']['gripper_width']['value'] = .02 + with self.assertRaises(ValueError): + validate_tape(tape) + + def test_native_jaw_controls_must_match_ramped_gap_symmetrically(self): + for controls in ([0.] * 8, [0.] * 6+[.02, .04], [0.] * 6+[.03, .03]): + tape = fixture() + tape['steps'][0]['applied']['native_position_controls'] = controls + with self.subTest(controls=controls), self.assertRaisesRegex(ValueError, 'half the recorded ramped gap'): + validate_tape(tape) + tape = fixture() + # Native joint controls include gravity compensation, so do not require + # them to equal ramped joint targets. + tape['steps'][0]['applied']['native_position_controls'][0] = .001 + validate_tape(tape) + + def test_native_clock_origin_matches_actual_declared_settling_ticks(self): + tape = fixture() + tape['clock']['native_time_offset_seconds'] = 987. + with self.assertRaisesRegex(ValueError, 'settling ticks'): + validate_tape(tape) + tape = fixture() + # This falls within the runtime's integer-ratio tolerance. Reset takes + # exactly 400 native ticks, rather than adopting the nominal time. + tape['configuration']['settle_time'] = .4000000000005 + tape['provenance']['configuration_sha256'] = digest(tape['configuration']) + validate_tape(tape) + + def test_extreme_finite_numeric_values_are_cleanly_refused_by_api_and_cli(self): + cases = [] + ratio = fixture() + ratio['configuration'].update(timestep=1e-308, control_dt=1e308, horizon=1e308) + ratio['provenance']['configuration_sha256'] = digest(ratio['configuration']) + cases.append(ratio) + large_finite_ratio = fixture() + large_finite_ratio['configuration'].update(timestep=1e-308, control_dt=1., horizon=1.) + large_finite_ratio['provenance']['configuration_sha256'] = digest(large_finite_ratio['configuration']) + cases.append(large_finite_ratio) + for quaternion in ([10**300, 0, 0, 0], [1e308, 1e308, 1e308, 1e308]): + orientation = fixture() + orientation['steps'][0]['after']['signals']['object_quaternion']['value'] = quaternion + cases.append(orientation) + with TemporaryDirectory() as directory: + for index, tape in enumerate(cases): + with self.subTest(index=index), self.assertRaises(ValueError): + validate_tape(tape) + path = Path(directory)/f'extreme-{index}.json' + path.write_bytes(canonical(tape)) + for operation in ('validate', 'inspect'): + result = subprocess.run([sys.executable, '-m', 'dvidia_training.episode_tape', operation, str(path)], + cwd=Path(__file__).resolve().parents[1], capture_output=True, text=True) + with self.subTest(index=index, operation=operation): + self.assertEqual(result.returncode, 1, result.stderr) + self.assertEqual(json.loads(result.stdout)['status'], 'error') + self.assertNotIn('Traceback', result.stderr) + + def test_terminal_single_native_tick_accepts_only_float_roundoff(self): + tape = fixture() + end = .00099999999999999 + tape['steps'][0]['after']['timestamp_seconds'] = end + for sensor in tape['steps'][0]['after']['signals'].values(): + sensor['captured_at_seconds'] = end + tape['steps'][0]['applied']['interval_end_seconds'] = end + tape['steps'][0]['result'].update(terminated=True, truncated=False, success=True, reason='success') + tape['outcome']['result'] = deepcopy(tape['steps'][0]['result']) + tape['outcome']['simulation_seconds'] = end + validate_tape(tape) + tape['steps'][0]['after']['timestamp_seconds'] = .0005 + for sensor in tape['steps'][0]['after']['signals'].values(): + sensor['captured_at_seconds'] = .0005 + tape['steps'][0]['applied']['interval_end_seconds'] = .0005 + tape['outcome']['simulation_seconds'] = .0005 + with self.assertRaises(ValueError): + validate_tape(tape) + + def test_false_success_scope_escalation_and_early_terminal_are_refused(self): + for field, value in [('success', True), ('terminated', True), ('valid', False), ('truncated', 1)]: + tape = fixture() + tape['steps'][0]['result'][field] = value + with self.subTest(field=field), self.assertRaises(ValueError): + validate_tape(tape) + for field, value in [('physical_robot_ready', True), ('schema_version', True), ('scope', 'hardware')]: + tape = fixture() + tape[field] = value + with self.subTest(field=field), self.assertRaises(ValueError): + validate_tape(tape) + tape = fixture() + tape['steps'].append(deepcopy(tape['steps'][0])) + with self.assertRaises(ValueError): + validate_tape(tape) + + def test_commands_cannot_be_silently_changed_or_recast_as_measurements(self): + tape = fixture() + tape['steps'][0]['command']['issued']['gripper_width'] = .01 + with self.assertRaises(ValueError): + validate_tape(tape) + tape = fixture() + tape['provenance']['case'] = 'open-jaw' + tape['steps'][0]['command']['policy_requested']['gripper_width'] = .01 + tape['steps'][0]['command']['intervention'] = 'open-jaw' + self.assertEqual(validate_tape(tape)['steps'][0]['after']['signals']['gripper_width']['value'], .08) + + def test_hashes_final_receipt_and_signal_contract_cannot_drift(self): + tape = fixture() + mutations = [lambda x: x['configuration'].update(object_mass=.08), + lambda x: x.update(protocol_sha256='0'*64), + lambda x: x['signal_contract']['object_position'].update(privileged=1), + lambda x: x['outcome'].update(control_steps=2), + lambda x: x['outcome'].update(simulation_seconds=.04)] + for mutation in mutations: + modified = deepcopy(tape) + mutation(modified) + with self.subTest(mutation=mutation), self.assertRaises(ValueError): + validate_tape(modified) + + def test_file_round_trip_is_fresh_and_cli_inspection_preserves_file(self): + with TemporaryDirectory() as directory: + path = Path(directory)/'episode.json' + receipt = write_tape(path, fixture()) + original = path.read_bytes() + self.assertEqual(receipt['bytes'], len(original)) + self.assertEqual(load_tape(path), fixture()) + with self.assertRaises(FileExistsError): + write_tape(path, fixture()) + output = io.StringIO() + with redirect_stdout(output): + self.assertEqual(main(['inspect', str(path)]), 0) + self.assertEqual(json.loads(output.getvalue())['unique_observations'], 2) + self.assertEqual(path.read_bytes(), original) + + def test_invalid_identifier_rejected_before_optional_native_import(self): + with self.assertRaises(ValueError): + record_episode(episode_id='../private', source_recording_id='s', session_id='g') + + +@unittest.skipUnless(find_spec('mujoco') is not None, 'Requires optional native arm extra') +class EpisodeTapeNativeTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.nominal = record_episode(episode_id='native-nominal', source_recording_id='fresh-native-01', session_id='fresh-session-19050', seed=19050) + cls.open_jaw = record_episode(episode_id='native-open-jaw', source_recording_id='fresh-native-02', session_id='fresh-session-19050', seed=19050, case='open-jaw') + cls.released = record_episode(episode_id='native-released', source_recording_id='fresh-native-03', session_id='fresh-session-19050', seed=19050, case='release-after-lift') + + def test_native_tape_preserves_command_measurement_and_terminal_semantics(self): + tape = self.nominal + self.assertTrue(tape['outcome']['result']['success']) + self.assertAlmostEqual(tape['clock']['native_time_offset_seconds'], .4, places=12) + steps = tape['steps'] + self.assertTrue(any(step['applied']['rate_limited'] for step in steps)) + self.assertTrue(any(abs(step['command']['issued']['gripper_width']-step['after']['signals']['gripper_width']['value']) > .002 for step in steps)) + for step in steps: + self.assertEqual(step['command']['issued_at_seconds'], step['before']['timestamp_seconds']) + self.assertEqual(step['after']['signals']['joint_position']['age_seconds'], 0.) + self.assertEqual(step['labels']['origin'], 'authored controller') + self.assertEqual(step['applied']['native_position_controls'][6:], [step['applied']['ramped_gripper_width']/2]*2) + self.assertTrue(tape['signal_contract']['object_position']['privileged']) + + def test_open_jaw_case_cannot_be_reported_as_successful_grasp(self): + tape = self.open_jaw + self.assertFalse(tape['outcome']['result']['success']) + self.assertEqual(tape['outcome']['result']['reason'], 'horizon') + self.assertEqual(inspect_tape(tape)['command_intervention_samples'], len(tape['steps'])) + self.assertTrue(all(step['after']['signals']['grasp_contacts']['value'] == [0, 0] for step in tape['steps'])) + self.assertEqual(tape['provenance']['scene_sha256'], self.nominal['provenance']['scene_sha256']) + + def test_released_case_records_intervention_and_authored_recovery_separately(self): + tape = self.released + summary = inspect_tape(tape) + self.assertTrue(tape['outcome']['result']['success']) + self.assertEqual(tape['outcome']['recovery_count'], 1) + self.assertGreater(summary['command_intervention_samples'], 0) + self.assertLessEqual(summary['command_intervention_samples'], 18) + self.assertGreater(summary['authored_action_phase_samples'].get('recover_wait', 0), 0) + intervened = [step for step in tape['steps'] if step['command']['intervention'] != 'none'] + self.assertTrue(any(step['command']['policy_requested']['gripper_width'] != .08 for step in intervened)) + self.assertTrue(all(step['command']['issued']['gripper_width'] == .08 for step in intervened)) + intervention_seconds = sum(step['applied']['interval_end_seconds']-step['applied']['interval_start_seconds'] for step in intervened) + self.assertAlmostEqual(intervention_seconds, .36, places=9) + self.assertEqual(tape['provenance']['scene_sha256'], self.nominal['provenance']['scene_sha256']) + self.assertEqual(tape['provenance']['session_id'], self.nominal['provenance']['session_id']) + + def test_native_record_validate_and_inspect_cli_round_trip(self): + with TemporaryDirectory() as directory: + path = Path(directory)/'native-cli.json' + prefix = [sys.executable, '-m', 'dvidia_training.episode_tape'] + command = prefix+['record', '--output', str(path), '--episode-id', 'native-cli', + '--source-recording-id', 'native-cli-recording', '--session-id', 'native-cli-session', + '--seed', '19050'] + recorded = subprocess.run(command, cwd=Path(__file__).resolve().parents[1], capture_output=True, text=True, timeout=60) + self.assertEqual(recorded.returncode, 0, recorded.stderr or recorded.stdout) + receipt = json.loads(recorded.stdout) + self.assertTrue(receipt['inspection']['outcome']['result']['success']) + original = path.read_bytes() + for operation in ('validate', 'inspect'): + result = subprocess.run(prefix+[operation, str(path)], cwd=Path(__file__).resolve().parents[1], capture_output=True, text=True, timeout=60) + self.assertEqual(result.returncode, 0, result.stderr or result.stdout) + self.assertEqual(json.loads(result.stdout)['scope'], 'simulation-only') + repeated = subprocess.run(command, cwd=Path(__file__).resolve().parents[1], capture_output=True, text=True, timeout=60) + self.assertEqual(repeated.returncode, 1, repeated.stderr) + self.assertEqual(json.loads(repeated.stdout)['status'], 'error') + self.assertEqual(path.read_bytes(), original) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_training_studio.py b/tests/test_training_studio.py index 71dd6bf..2d1af23 100644 --- a/tests/test_training_studio.py +++ b/tests/test_training_studio.py @@ -11,9 +11,11 @@ import threading import time import unittest +from unittest import mock import zipfile -from dvidia_training.training_studio import TrainingStudioServer, UPLOAD_LIMIT, configure +from dvidia_training.coverage import MAX_PLAN_BYTES +from dvidia_training.training_studio import JSON_LIMIT, TrainingStudioServer, UPLOAD_LIMIT, configure def artifact_run(source, output, *, offline=True, seed=17): @@ -135,6 +137,146 @@ def test_initial_ui_and_state_do_not_claim_training_or_robot_readiness(self): self.assertEqual(self.request('/download', method='GET')[0], 409) self.assertEqual(self.request('/api/result', method='GET')[0], 409) + def test_coverage_ui_and_example_are_planning_only_and_create_no_job(self): + status, headers, body = self.request('/coverage', method='GET') + self.assertEqual(status, 200) + self.assertIn(b'Cover the skill, one useful example at a time.', body) + self.assertIn(b'Not evaluated', body) + self.assertIn(b'Not run', body) + self.assertIn(b'No footage has been checked', body) + self.assertIn("frame-ancestors 'none'", headers['Content-Security-Policy']) + self.assertEqual(headers['X-Content-Type-Options'], 'nosniff') + self.assertNotIn('Access-Control-Allow-Origin', headers) + status, _, plan = self.request('/api/coverage/example', method='GET') + self.assertEqual(status, 200) + self.assertEqual(plan['format'], 'dvidia.skillspace-coverage-plan') + self.assertTrue(plan['plan_id'].startswith('demo-')) + status, _, summary = self.request('/api/coverage/inspect', plan) + self.assertEqual(status, 200, summary) + self.assertEqual(summary['plan'], plan) + self.assertEqual(summary['coverage']['cells_met'], 2) + self.assertEqual(summary['coverage']['cells_total'], 6) + self.assertFalse(summary['coverage']['all_cells_met']) + self.assertFalse(summary['robot_skill_acquired']) + self.assertFalse(summary['physical_robot_ready']) + self.assertFalse(summary['benchmark']['executed']) + self.assertFalse(summary['benchmark']['qualified']) + self.assertFalse(summary['recognition']['payments_active']) + self.assertFalse(summary['recognition']['awards_issued']) + self.assertTrue(any('fabricated' in warning for warning in summary['warnings'])) + self.assertTrue(any('declared metadata' in warning for warning in summary['warnings'])) + self.assertIsNone(self.server.job) + self.assertFalse((self.server.directory / 'jobs').exists()) + self.assertFalse((self.server.directory / 'uploads').exists()) + self.assertEqual(self.outcome()['status'], 'idle') + + def test_coverage_routes_keep_the_loopback_origin_and_host_boundary(self): + plan = self.request('/api/coverage/example', method='GET')[2] + port = self.server.server_address[1] + for path, method, value in (('/coverage', 'GET', None), + ('/api/coverage/example', 'GET', None), + ('/api/coverage/inspect', 'POST', plan)): + for headers in ({'Host': 'evil.example'}, {'Host': f'localhost:{port + 1}'}, + {'Origin': 'https://evil.example'}, {'Origin': 'null'}, + {'Sec-Fetch-Site': 'cross-site'}): + with self.subTest(path=path, headers=headers): + status, _, result = self.request(path, value, method=method, headers=headers) + self.assertEqual(status, 400) + self.assertIn('error', result) + self.assertIsNone(self.server.job) + self.assertFalse((self.server.directory / 'jobs').exists()) + + def test_coverage_inspection_rejects_malformed_forged_and_oversized_json(self): + plan = self.request('/api/coverage/example', method='GET')[2] + forged_benchmark = json.loads(json.dumps(plan)) + forged_benchmark['benchmark']['qualified'] = True + forged_skill = json.loads(json.dumps(plan)) + forged_skill['robot_skill_acquired'] = True + duplicated_plan = json.dumps(plan).replace('"plan_revision": 1', '"plan_revision": 1, "plan_revision": 1', 1) + cases = [('[]', {}), ('{}', {}), ('{"plan_id":"a","plan_id":"b"}', {}), + (duplicated_plan, {}), + ('{"plan_revision":NaN}', {}), ('{"plan_revision":1e999}', {}), + (json.dumps(forged_benchmark), {}), (json.dumps(forged_skill), {}), + (json.dumps({'source': '/etc/passwd'}), {}), + (json.dumps(plan), {'Content-Type': 'text/plain'}), + ('{}', {'Content-Length': str(MAX_PLAN_BYTES + 1)}), + ('{}', {'Transfer-Encoding': 'chunked'})] + for raw, headers in cases: + with self.subTest(raw=raw[:80], headers=headers): + status, _, result = self.request('/api/coverage/inspect', raw=raw, headers=headers) + self.assertEqual(status, 400, result) + self.assertIn('error', result) + if raw == duplicated_plan: + self.assertIn('Duplicate JSON keys', result['error']) + self.assertIsNone(self.server.job) + self.assertFalse((self.server.directory / 'jobs').exists()) + self.assertFalse((self.server.directory / 'uploads').exists()) + + def test_large_coverage_metadata_roundtrips_without_relaxing_training_limit(self): + plan = self.request('/api/coverage/example', method='GET')[2] + original = plan['evidence'][0] + plan['evidence'] = [] + for index in range(100): + identity = f'http-test-{index}' + plan['evidence'].append({**original, 'id': identity, + 'media_sha256': hashlib.sha256(identity.encode()).hexdigest(), + 'source_recording_id': identity + '-recording', + 'session_id': identity + '-session', + 'object_instance_id': identity + '-object'}) + goal = plan['skill']['goal'] + plan['skill']['goal'] = ' ' + goal + ' ' + raw = json.dumps(plan) + self.assertGreater(len(raw.encode()), JSON_LIMIT) + self.assertLess(len(raw.encode()), MAX_PLAN_BYTES) + status, _, summary = self.request('/api/coverage/inspect', raw=raw) + self.assertEqual(status, 200, summary) + self.assertEqual(summary['plan']['skill']['goal'], goal) + self.assertEqual(summary['plan']['evidence'], plan['evidence']) + self.assertEqual(summary['coverage']['submitted_attempts'], 100) + self.assertEqual(summary['coverage']['accepted_groups'], 100) + self.assertFalse(summary['robot_skill_acquired']) + self.assertFalse(summary['benchmark']['executed']) + self.assertFalse(summary['benchmark']['qualified']) + # The planner's metadata ceiling must not expand command/job requests. + status, _, error = self.request('/api/train', raw=json.dumps({'source': '/local/export'}) + ' ' * JSON_LIMIT) + self.assertEqual(status, 400) + self.assertIn('error', error) + self.assertIn('32 KiB limit', error['error']) + self.assertIsNone(self.server.job) + self.assertFalse((self.server.directory / 'jobs').exists()) + self.assertFalse((self.server.directory / 'uploads').exists()) + + def test_coverage_inspection_does_not_open_media_or_submit_work(self): + plan = self.request('/api/coverage/example', method='GET')[2] + # Import the planner before checking its request-time filesystem boundary. + self.assertEqual(self.request('/api/coverage/inspect', plan)[0], 200) + with mock.patch.object(Path, 'open', side_effect=AssertionError('Planning must not open a file.')), \ + mock.patch.object(TrainingStudioServer, 'submit', side_effect=AssertionError('Planning must not submit a job.')): + self.assertEqual(self.request('/api/coverage/inspect', plan)[0], 200) + self.assertEqual(self.request('/api/coverage/inspect', {'source': '/etc/passwd'})[0], 400) + self.assertIsNone(self.server.job) + self.assertFalse((self.server.directory / 'jobs').exists()) + + def test_valid_and_invalid_coverage_plans_cannot_replace_a_completed_result(self): + self.assertEqual(self.train()['status'], 'complete') + job = self.server.job + archive = self.request('/download', method='GET')[2] + state = self.outcome() + plan = self.request('/api/coverage/example', method='GET')[2] + forged = json.loads(json.dumps(plan)) + forged['benchmark']['executed'] = True + for value, expected_status in ((plan, 200), (forged, 400), ({}, 400), + ({'source': '/local/replacement'}, 400)): + with self.subTest(expected_status=expected_status, fields=list(value)): + self.assertEqual(self.request('/api/coverage/inspect', value)[0], expected_status) + self.assertIs(self.server.job, job) + self.assertEqual(self.outcome(), state) + status, _, downloaded = self.request('/download', method='GET') + self.assertEqual(status, 200) + self.assertEqual(downloaded, archive) + self.assertEqual(self.request('/coverage', method='GET')[0], 200) + self.assertIs(self.server.job, job) + def test_browser_origin_and_host_guards_cannot_replace_a_completed_result(self): self.assertEqual(self.train()['status'], 'complete') port = self.server.server_address[1] From e2749927248d1d500a44add3c2c7084230c2fbdc Mon Sep 17 00:00:00 2001 From: DVIDIA Date: Fri, 9 Oct 2026 05:59:02 -0400 Subject: [PATCH 2/2] Use installed episode tape commands in the guide --- docs/episode-tape.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/episode-tape.md b/docs/episode-tape.md index 198d23a..877f53e 100644 --- a/docs/episode-tape.md +++ b/docs/episode-tape.md @@ -14,26 +14,29 @@ have not been established across machines. ## Record fresh evidence Install the simulator extra using the [installation guide](installation.md). +The commands below run from the checkout with its installed virtual environment. +In Windows PowerShell, replace `.venv/bin/COMMAND` with `.\venv\Scripts\COMMAND.exe` +and put continued commands on one line. Use a fresh output path outside the frozen benchmark directories: ```sh mkdir -p runs/episode-tapes -python -m dvidia_training.episode_tape record \ +.venv/bin/dvidia-episode-tape record \ --output runs/episode-tapes/nominal-19050.json \ --episode-id nominal-19050 \ --source-recording-id recording-nominal-19050 \ --session-id simulation-session-19050 \ --seed 19050 --case nominal -python -m dvidia_training.episode_tape record \ +.venv/bin/dvidia-episode-tape record \ --output runs/episode-tapes/open-jaw-19050.json \ --episode-id open-jaw-19050 \ --source-recording-id recording-open-jaw-19050 \ --session-id simulation-session-19050 \ --seed 19050 --case open-jaw -python -m dvidia_training.episode_tape validate runs/episode-tapes/nominal-19050.json -python -m dvidia_training.episode_tape inspect runs/episode-tapes/open-jaw-19050.json +.venv/bin/dvidia-episode-tape validate runs/episode-tapes/nominal-19050.json +.venv/bin/dvidia-episode-tape inspect runs/episode-tapes/open-jaw-19050.json ``` The `nominal` case records the unchanged authored controller. `open-jaw` keeps