diff --git a/README.md b/README.md
index b54ffd2..368bc46 100644
--- a/README.md
+++ b/README.md
@@ -2,13 +2,13 @@
**[English](README.md)** · [简体中文](README.zh-CN.md)
-**Run robot manipulation experiments locally, and inspect the full visual control loop.** ManiLoop connects GPT or learned policies to MuJoCo tasks through a browser workspace and CLI: choose a task, connect a model, execute actions, and review what actually happened.
+**Closed-loop robot policy evaluation across simulation benchmarks.** Designed to support fair, reproducible comparisons across policies and task environments, ManiLoop brings built-in MuJoCo scenes, LIBERO, robosuite, and RoboCasa into one local browser workspace and CLI. Start with manual control, then use GPT, a compatible vision API, a supported local LeRobot checkpoint, or Jev text primitives.
**[Get started ↓](#step-1--open-the-local-workspace)** · [Connect a model](#step-3--connect-your-model) · [Observed results](docs/RESULTS.md#english) · [Project website](https://daniel-rmc.github.io/ManiLoop/?lang=en)
-
+
-*The actual local application, shown in its current Chinese interface. This is the working simulator UI, not the website’s recorded replay.*
+*The current local application in its Chinese interface, previewing the built-in Panda pick-and-place scene. These controls operate the simulator; the project website separately offers recorded episode playback.*
[Watch GPT place a bowl, turn a stove knob, and move cream cheese into a basket](https://daniel-rmc.github.io/ManiLoop/?lang=en#skills): play verified episodes, pause at each action, and read the model's recorded action notes alongside the controller feedback. Full videos and execution conditions are included.
@@ -16,23 +16,28 @@ These three recorded GPT successes took **5.5–11.1 minutes of episode wall tim
## What you can run
-The workspace brings **camera observations, model actions, controller feedback, and independent evaluation** together. Use manual controls before connecting a model, single-step GPT decisions, pause and resume, compare before/after observations, run CLI experiment matrices, and export complete LIBERO episodes.
+The workspace brings **camera observations, actions, controller feedback, and independent evaluation** together. Use manual controls, single-step model decisions, pause and resume, compare before/after observations, and run CLI experiment matrices. Complete control-step video capture is available for LIBERO in controlled timing; other backends provide decision logs and review frames.
-| Environment / task | Available operations |
-| --- | --- |
-| Built-in `pick_place` | Lift the red cube, place it inside the target region, release, and settle |
-| Built-in `push` | Push the cube along the table into the target region without lifting it |
-| `libero_spatial` · 10 tasks | Select and move a bowl based on its spatial relationship to other objects |
-| `libero_object` · 10 tasks | Pick different objects and place them in a basket |
-| `libero_goal` · 10 tasks | Goal variants including drawer opening, object placement, plate pushing, and stove control |
-| `libero_90` · 90 tasks | Kitchen, living-room, and study tasks, including drawers, microwaves, and stacking |
-| `libero_10` · 10 tasks | Longer tasks combining multiple objects or manipulation operations |
+| Backend | Robot and available tasks | Integration scope |
+| --- | --- | --- |
+| Built-in MuJoCo | ARX X5 / Franka Panda; `pick_place` and `push`; `tabletop_a` / `tabletop_b` | Contact-based tabletop manipulation, manual and model control, independent task evaluation |
+| [LIBERO](docs/LIBERO.md) | Official Panda; five task suites | Official initial states, OSC controller, and success rules; selected real model episodes are published below |
+| [robosuite](docs/ROBOSUITE.md) | Panda; `Lift`, `Stack`, `PickPlaceCan`, `Door`, `NutAssemblySquare` | Five integrated tasks with camera, stepping, reset, and evaluation checks; world-frame OSC adaptation |
+| [RoboCasa](docs/ROBOCASA.md) | PandaOmron; `OpenDrawer`, `CloseDrawer`, `OpenCabinet`, `CoffeeSetupMug` | Four kitchen tasks; arm and gripper control, zero base velocity and torso delta; no navigation |
-The built-in tasks support **ARX X5 and Franka Panda**, each with `tabletop_a` and `tabletop_b` layouts. LIBERO uses its official Panda, initial states, OSC controller, and success rules. These are available task catalogs; they do not mean every task has been completed by every model. Changing the instruction text does not change a task’s evaluator.
+LIBERO's catalogs contain **10 spatial, 10 object, 10 goal, 90 `libero_90`, and 10 `libero_10` tasks**, covering object placement, drawers, stove control, stacking, and longer operations. These are catalog sizes, not counts of tasks completed by a model. RoboCasa's four-task interface checks use layout 11 / style 14 / seed 0 on macOS / Apple Silicon; other scene combinations and arm reachability need separate checks. Changing the instruction text does not change an upstream evaluator.
-Policies receive images and robot sensor state. **Object ground truth, simulator contacts, rewards, and success signals are excluded from policy input.** Independent evaluation can inspect simulator state and stop an episode; a model’s `done` declaration is not the success criterion.
+Vision policies receive images and robot sensor state; **Jev receives explicit text and proprioception only**. Object ground truth, simulator contacts, rewards, and success signals are excluded from every policy input. Independent evaluation can inspect simulator state and stop an episode; `done`, controller `reached`, and completion of a command sequence are distinct from task success.
-[See real manual debugging: arm and gripper controls, sensor readings, execution feedback, and independent evaluation](docs/images/workspace-feedback.png). This screenshot shows feedback from a manual jog, not a successful task.
+The local workspace has three panels:
+
+| Panel | What you do |
+| --- | --- |
+| Manual control | Translate and rotate the TCP in fixed or tool axes, open or close the gripper, and submit a combined six-dimensional increment. Inspect position and orientation errors. |
+| Model execution | Enter instructions, choose a connection, run diagnostics, and use single-step, continuous execution, or pause/resume. |
+| Scene settings | Choose a backend, task, robot, and scene parameters; apply the selection to rebuild the environment. |
+
+Stop remains visible across panels. Switching panels or between picture-in-picture and side-by-side cameras does not start inference or advance physics. Manual control requires the current task, action, and request to have finished. The manual target-tracking budget is separate from the model budget; TCP rotations are not individual-joint commands. See [workspace usage](docs/WORKSPACE.md) and [six-dimensional manual control](docs/MANUAL_CONTROL.md).
## Step 1 · Open the local workspace
@@ -59,35 +64,65 @@ py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m maniloop demo
```
-Open [http://127.0.0.1:8765](http://127.0.0.1:8765). Select a robot, layout, and task; apply the configuration, try manual arm and gripper controls, then reset. **No API key is needed for this step.** The server listens on localhost; keep its terminal open and press `Ctrl+C` to stop it. Use `--port 8766` if needed.
+Open [http://127.0.0.1:8765](http://127.0.0.1:8765). The workspace starts in Manual control. Use Scene settings to select a robot, layout, and task, apply the configuration, then return to manual arm and gripper controls. **No API key is needed for this step.** The server listens on localhost; keep its terminal open and press `Ctrl+C` to stop it. Use `--port 8766` if needed.
-Continue from the repository root after stopping the server. **Windows:** replace subsequent `python` commands with `.\.venv\Scripts\python.exe`; join multiline Bash examples into one line, removing trailing `\`. After installing `uv`, add `--uv .\.venv\Scripts\uv.exe` when running either setup script. No activation script or execution-policy change is required.
+Continue from the repository root after stopping the server. **Windows:** replace subsequent `python` commands with `.\.venv\Scripts\python.exe`; join multiline Bash examples into one line, removing trailing `\`. After installing `uv`, add `--uv .\.venv\Scripts\uv.exe` when running a setup script. No activation script or execution-policy change is required.
For headless Ubuntu/Debian, install `libosmesa6` and set `MUJOCO_GL=osmesa` before starting Python. EGL is another option with compatible drivers. These Linux settings do not apply to macOS.
-## Step 2 · Add official LIBERO tasks
+## Step 2 · Choose a simulation backend
+
+The built-in scenes are ready after Step 1. Choose the optional backend you need from the sections below and run its commands from the repository root. Stop a running demo before following another launch example. Workers use separate interpreters; continue launching the workspace with the main environment's `python`.
```bash
python -m pip install uv
+```
+
+### LIBERO
+
+```bash
python scripts/setup_libero.py
python -m maniloop list --backend libero --libero-suite libero_goal
python -m maniloop smoke --backend libero
python -m maniloop demo --backend libero --port 8767
```
-Open [http://127.0.0.1:8767](http://127.0.0.1:8767). Select a suite, task ID, and initialization ID in the environment panel. IDs start at zero; `list` prints the available task names. The default is `libero_spatial` task 0, init 0: move the black bowl between the plate and ramekin onto the plate.
+Open [http://127.0.0.1:8767](http://127.0.0.1:8767). Select a suite, task ID, and initialization ID in Scene settings. IDs start at zero; `list` prints the available task names. The default is `libero_spatial` task 0, init 0: move the black bowl between the plate and ramekin onto the plate. Installation needs network access and several GB of disk space, but no demonstration dataset or policy weights. See [LIBERO setup](docs/LIBERO.md).
-The installer needs network access and several GB of disk space, but no demonstration dataset or policy weights. It downloads the pinned official source, assets, and a managed interpreter while keeping dependencies separate:
+### robosuite
+
+```bash
+python scripts/setup_robosuite.py
+python -m maniloop list --backend robosuite
+python -m maniloop demo --backend robosuite --task Lift --port 8870
+```
+
+Open [http://127.0.0.1:8870](http://127.0.0.1:8870). The five Panda tasks use robosuite 1.5.2 / MuJoCo 3.3.7 in their own runtime. This adapter explicitly configures world-frame OSC and records that choice. Camera, manual control, and model interfaces are integrated; no model success rate is established for these tasks. See [control semantics and checks](docs/ROBOSUITE.md).
+
+### RoboCasa
+
+```bash
+python scripts/setup_robocasa.py --download-assets
+python -m maniloop list --backend robocasa
+python -m maniloop demo --backend robocasa --task OpenDrawer \
+ --robocasa-layout 11 --robocasa-style 14 --port 8872
+```
+
+Open [http://127.0.0.1:8872](http://127.0.0.1:8872). The installer pins RoboCasa and its compatible robosuite source separately from the standalone robosuite backend. Its five required asset groups occupy about 11.1 GB unpacked, plus source/runtime space. PandaOmron's arm and gripper are controlled while the base receives zero velocity and the torso receives zero delta; this is not a navigation interface. Default-scene interface verification does not establish autonomous task success or arm-only solvability in every scene. See [RoboCasa setup and limitations](docs/ROBOCASA.md).
+
+### Keep runtimes separate
| Environment | Purpose |
| --- | --- |
| `.venv` · Python 3.12 | Main application, UI, model connections, bundled MuJoCo scenes |
| `.venv-libero` · Python 3.10 | LIBERO simulation and its pinned dependencies |
+| `.venv-robosuite` · Python 3.11 | Standalone robosuite simulation |
+| `.venv-robocasa` · Python 3.11 | RoboCasa simulation and its compatible upstream dependencies |
| `.venv-vla` · Python 3.12 | Optional LeRobot inference and learned policies |
-Keep these environments separate. See [LIBERO setup and task selection](docs/LIBERO.md).
+Do not combine these dependencies into the main environment. The RoboCasa worker is a simulation installation, not the upstream training environment. Optional assets, environments, and model weights are downloaded locally and are not bundled in this repository.
## Step 3 · Connect your model
@@ -98,12 +133,13 @@ Choose one route below. Stop the existing server before running a different laun
| GPT through Codex | Official Codex CLI login; select the Codex source | Uses models available to that CLI/account; ManiLoop does not extract login tokens |
| Hosted or self-hosted vision model | Enter the service base URL, model ID, and API key, or import provider TOML | Robot control requires **Responses API + image input + JSON Schema output** |
| Local LeRobot checkpoints | Select a supported model loaded by the isolated inference worker | Three reviewed LIBERO checkpoint presets; not arbitrary model-folder import |
+| Jev text primitives | Enter a separate TypeSafe key in the model panel | Explicit text and robot proprioception; bounded arm/gripper commands, without images or object localization |
See model controls in the actual local application
-
+
-*Current local UI; model and task settings here configure real execution.*
+*The current Model execution panel in its Chinese interface, with the built-in Panda scene loaded and no model request started. Its connections and instructions configure real execution; this is separate from the website's saved-data playground.*
### GPT / compatible model APIs
@@ -115,7 +151,7 @@ codex login
python -m maniloop demo --backend libero --codex-login --port 8767
```
-Choose the Codex source and load the GPT demo preset. Start with the text/image/action-format diagnostics or a single action. `MANILOOP_CODEX_BIN` selects the executable if several CLI versions are installed. See [GPT controls](docs/GPT6_DEMO.md).
+Choose the Codex source and load the GPT demo preset. This preset selects LIBERO; to work in another backend, keep that scene selected and configure the model directly. Start with the text/image/action-format diagnostics or a single action. `MANILOOP_CODEX_BIN` selects the executable if several CLI versions are installed. See [GPT controls](docs/GPT6_DEMO.md).
For an API connection, select the manual or imported-config source, enter your model ID and the provider’s base URL, then supply its key. Examples: [OpenAI configuration](examples/openai.example.toml), [custom provider configuration](examples/custom.example.toml). A model you host locally can use this route **if its server implements the required Responses protocol**; a local base URL can be `http://127.0.0.1:8000/v1`. A Chat Completions-only server is not compatible with robot control; the separate [text chat page](docs/API_CHAT.md) supports both protocols. Selecting a model ID does not guarantee it supports vision or structured actions. Requests consume the selected account quota or provider billing; opening the page does not start inference.
@@ -126,7 +162,7 @@ python scripts/setup_vla.py --models smolvla-libero act-libero diffusion-libero
python -m maniloop demo --backend libero --port 8767
```
-Select the local LeRobot source and a downloaded model. Omit `--models` to install only SmolVLA. Inference runs offline on CPU, Apple MPS, or CUDA; the checkpoint’s saved preprocessing and action queue are retained.
+Select the local LeRobot source and a downloaded model. Omit `--models` to install only SmolVLA. These presets remain **LIBERO-only**, including when other backends are installed. Inference runs offline on CPU, Apple MPS, or CUDA; the checkpoint’s saved preprocessing and action queue are retained.
Or run a local checkpoint from the CLI; replace `smolvla-libero` with `act-libero` or `diffusion-libero` to select another installed preset:
@@ -145,6 +181,20 @@ python -m maniloop benchmark --backend libero --agent lerobot \
To use these presets **already stored on another local disk**, set `MANILOOP_MODELS_ROOT` to their parent directory. It must contain the preset-named folders, such as `smolvla-libero/`, plus `smolvlm-tokenizer/` for SmolVLA. `MANILOOP_VLA_PYTHON` can point to a compatible inference environment. These options relocate supported snapshots; they do not import arbitrary Hugging Face repositories or custom weights. Details and fixed revisions: [local model guide](docs/VLA.md).
+### Jev: explicit text commands
+
+In Model execution, select **Jev · 文本原语控制** and enter a TypeSafe API key. The key is separate from OpenAI connections and stays in server memory, not files or browser storage. Saving it does not call the model; the connection diagnostic and command execution use the TypeSafe API. No Jev weights or TypeSafe SDK installation is needed.
+
+Use one action per line, for example:
+
+```text
+Move up along world Z by 10 mm.
+Rotate about the tool Z axis by 5 degrees.
+Open the gripper.
+```
+
+Jev selects a bounded primitive from text and robot proprioception; local code handles units, frames, and execution. It receives no camera images and cannot locate an object for autonomous grasping. Controlled timing is required. A sequence supports up to 20 lines and stops on ambiguous commands, low candidate probability, API errors, or failed control. Its existing simulation checks use mocked Jev responses; real service behavior requires a separate diagnostic or command run. See [Jev setup and verification scope](docs/JEV.md).
+
### Integrating your own checkpoint or policy
A new checkpoint needs its own model registration, correct preprocessing and action mapping, and source metadata. Do not replace a preset’s files and treat the result as that preset. For a custom inference implementation, implement [`Agent`](src/maniloop/agents/base.py): `reset()` and `decide(task, observation, images, history, geometry_results)`, returning an action dictionary or [`ActionChunk`](src/maniloop/core/actions.py). Connect it to `EpisodeRunner`, and add a CLI/UI choice if needed; there is no automatic arbitrary-checkpoint loader.
@@ -153,7 +203,7 @@ A new simulation backend implements [`Environment`](src/maniloop/backends/base.p
## Step 4 · Record a run and inspect its result
-Run a new GPT-6 attempt from official initialization, with all LIBERO control steps recorded:
+For ordinary model runs, inspect the decision log, controller feedback, before/after frames, and independent result. Complete control-step recording and video export currently require **LIBERO with controlled timing**. The following runs a new GPT-6 attempt from official initialization and records every control step:
```bash
python -m maniloop benchmark \
@@ -191,7 +241,7 @@ Each row is **one selected run** on `libero_spatial` task 0 / init 0 / seed 0, n
## Guides, checks, and scope
-[LIBERO tasks](docs/LIBERO.md) · [GPT controls](docs/GPT6_DEMO.md) · [Local checkpoints](docs/VLA.md) · [API text chat](docs/API_CHAT.md) · [Recording](docs/EPISODE_RECORDING.md) · [Architecture](docs/ARCHITECTURE.md)
+[Workspace](docs/WORKSPACE.md) · [Manual control](docs/MANUAL_CONTROL.md) · [LIBERO](docs/LIBERO.md) · [robosuite](docs/ROBOSUITE.md) · [RoboCasa](docs/ROBOCASA.md) · [GPT controls](docs/GPT6_DEMO.md) · [Jev](docs/JEV.md) · [Local checkpoints](docs/VLA.md) · [API text chat](docs/API_CHAT.md) · [Recording](docs/EPISODE_RECORDING.md) · [Architecture](docs/ARCHITECTURE.md)
```bash
python -m pip install -r requirements-dev.txt
@@ -199,59 +249,10 @@ python -m pytest -q
python -m maniloop smoke
```
-These checks do not make model requests. Offline tests cover macOS, Linux, and Windows; rendering has been verified on macOS and Linux OSMesa. Windows rendering and LIBERO execution have not been validated. Rendering needs a graphics backend. ManiLoop is simulation software: the ARX gripper is approximate, and real-robot control or safety has not been validated. Official LIBERO assets alone do not establish paper-level benchmark comparability.
+These checks do not make model requests. Offline tests cover macOS, Linux, and Windows; main-application rendering has been verified on macOS and Linux OSMesa. Optional backends have their own opt-in integration checks in the linked guides; ordinary pytest does not validate an uninstalled backend. Rendering and LIBERO execution on Windows have not been validated. Rendering needs a graphics backend. ManiLoop is simulation software: the ARX gripper is approximate, and real-robot control or safety has not been validated. Environment integration, controller checks, and real model task outcomes are reported separately.
## License
-ManiLoop is [MIT licensed](LICENSE). ARX X5 assets retain MIT; Panda assets retain Apache-2.0. LIBERO, LeRobot, and downloaded checkpoints retain their own licenses. See [third-party notices](THIRD_PARTY_NOTICES.md).
-
-## Optional robosuite tasks
-
-Add Panda Lift, Stack, PickPlaceCan, Door, and NutAssemblySquare in an isolated
-robosuite runtime, without changing LIBERO. Use the existing workspace for manual
-control, model connections, and decision review. See [setup and limitations](docs/ROBOSUITE.md).
-
-```bash
-python scripts/setup_robosuite.py
-python -m maniloop demo --backend robosuite --task Lift --port 8870
-```
-
-These are integrated environments, not model-success claims. Existing LIBERO-trained
-local checkpoint presets remain LIBERO-only; RoboCasa is not included in this backend.
-
-## Optional RoboCasa kitchen tasks
-
-Four kitchen tasks are available through a separate PandaOmron backend: `OpenDrawer`,
-`CloseDrawer`, `OpenCabinet`, and `CoffeeSetupMug`. The first integration controls the
-arm and gripper; base velocity and torso delta inputs are held at zero.
-
-```bash
-python scripts/setup_robocasa.py --download-assets
-python -m maniloop demo --backend robocasa --task OpenDrawer --port 8872
-```
-
-Use the same workspace for camera observations, manual controls, model connections,
-and decision review. The installer pins both source projects and isolates their dependencies.
-The five required asset groups occupy about 11.1 GB unpacked, excluding source/runtime files.
-Default layout 11 / style 14 / seed 0 has real interface checks for all four tasks;
-this is not a model success result or a guarantee of arm-only task solvability in every scene.
-See [RoboCasa setup, action semantics, and limitations](docs/ROBOCASA.md).
-
-
-### Operator workspace
-
-The workspace separates manual control, model execution, and scene settings. Stop stays available while switching panels; camera layouts do not advance physics. See [workspace usage](docs/WORKSPACE.md) and [six-dimensional manual control](docs/MANUAL_CONTROL.md).
-
-### Jev text commands
-
-Choose **Jev · 文本原语控制** in the model panel to map explicit, one-line commands
-to bounded arm or gripper primitives. Enter a separate TypeSafe API key in the workspace;
-it is kept in server memory and is not shared with OpenAI connections. Saving the key
-does not call the model; connection diagnostics and execution use the TypeSafe API.
-
-Jev receives text and robot proprioception, without camera images or object state.
-This mode supports commands such as moving 10 mm or rotating the tool 5 degrees;
-it does not autonomously locate or grasp objects. Completing the command sequence
-is distinct from task success. See [Jev setup, examples, and verification](docs/JEV.md).
+ManiLoop is [MIT licensed](LICENSE). ARX X5 assets retain MIT; Panda assets retain Apache-2.0. LIBERO, robosuite, RoboCasa, LeRobot, and downloaded assets and checkpoints retain their own licenses. See [third-party notices](THIRD_PARTY_NOTICES.md) and [contributing](CONTRIBUTING.md).
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 671975a..6c2717e 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -2,13 +2,13 @@
[English](README.md) · **[简体中文](README.zh-CN.md)**
-**在本地运行机器人操作实验，看清完整的视觉控制过程。** ManiLoop 通过浏览器工作台与命令行，将 GPT 或学习策略接入 MuJoCo 任务：选择任务、连接模型、执行动作，再核对实际发生了什么。
+**面向跨策略、跨基准的机器人闭环评测。** ManiLoop 以公平、可复现的比较为目标，将内置 MuJoCo 场景、LIBERO、robosuite 和 RoboCasa 接入同一个本地浏览器工作台与命令行。从手动控制开始，再选择 GPT、兼容的视觉模型 API、受支持的本地 LeRobot 检查点，或 Jev 文本原语。
**[开始使用 ↓](#第一步--打开本地工作台)** · [连接模型](#第三步--连接模型) · [实测结果](docs/RESULTS.md#简体中文) · [项目网站](https://daniel-rmc.github.io/ManiLoop/?lang=zh)
-
+
-*本机实际运行的工作台，展示当前中文界面。这是可操作的仿真程序，不是网站上的录制回放。*
+*当前本地程序的中文界面，预览内置 Panda 抓放场景。这里的控件用于操作仿真；项目网站另提供已录制 episode 的回放。*
[观看 GPT 放碗、转动炉具旋钮、将奶油奶酪放入篮子](https://daniel-rmc.github.io/ManiLoop/?lang=zh#skills)：播放经过核验的回合，逐步暂停，对照模型当时的动作说明与控制器反馈，并查看完整录像和执行条件。
@@ -16,23 +16,28 @@
## 可以运行哪些任务
-工作台将**相机观测、模型动作、控制器反馈与独立评分**放在一起。你可以先手动调试，再连接模型；逐步执行 GPT 决策、暂停和继续、对照执行前后观测，通过命令行批量运行实验，并导出完整 LIBERO episode。
+工作台将**相机观测、动作、控制器反馈与独立评分**放在一起。你可以手动调试、逐步执行模型决策、暂停和继续、对照执行前后观测，并通过命令行批量运行实验。完整逐控制步录像目前支持受控时序下的 LIBERO；其他后端提供决策日志与回看画面。
-| 环境／任务 | 可执行的操作 |
-| --- | --- |
-| 内置 `pick_place` | 抬起红色方块，放入目标区域，松爪后等待稳定 |
-| 内置 `push` | 沿桌面将方块推入目标区域，不抬起方块 |
-| `libero_spatial` · 10 个任务 | 根据物体间的空间关系选择并移动碗 |
-| `libero_object` · 10 个任务 | 抓取不同物体并放入篮子 |
-| `libero_goal` · 10 个任务 | 开抽屉、放置物体、推盘子、操作炉具等目标变化 |
-| `libero_90` · 90 个任务 | 厨房、客厅与书房操作，包括抽屉、微波炉和堆叠 |
-| `libero_10` · 10 个任务 | 涉及多个物体或多项操作的较长任务 |
+| 后端 | 机器人与可用任务 | 接入范围 |
+| --- | --- | --- |
+| 内置 MuJoCo | ARX X5／Franka Panda；`pick_place` 抓放、`push` 推物；`tabletop_a`／`tabletop_b` 布局 | 基于接触物理的桌面操作、手动与模型控制、独立任务评分 |
+| [LIBERO](docs/LIBERO.md) | 官方 Panda；五个任务套件 | 官方初始状态、OSC 控制器与成功规则；下文展示选定的真实模型回合 |
+| [robosuite](docs/ROBOSUITE.md) | Panda；`Lift`、`Stack`、`PickPlaceCan`、`Door`、`NutAssemblySquare` | 五项任务已接入相机、步进、重置与评分检查；采用世界系 OSC 适配 |
+| [RoboCasa](docs/ROBOCASA.md) | PandaOmron；`OpenDrawer`、`CloseDrawer`、`OpenCabinet`、`CoffeeSetupMug` | 四项厨房任务；控制机械臂与夹爪，底座速度和躯干增量为零；不含导航 |
-内置任务支持 **ARX X5 和 Franka Panda**，均可选择 `tabletop_a`、`tabletop_b` 两种布局。LIBERO 使用官方 Panda、初始状态、OSC 控制器与成功规则。表中列出的是可用任务目录，不代表每个模型都已完成全部任务。修改文字指令不会改变任务的评分规则。
+LIBERO 目录包括 **spatial 10 项、object 10 项、goal 10 项、`libero_90` 90 项、`libero_10` 10 项**，覆盖物体放置、抽屉、炉具、堆叠和较长操作。这里是目录规模，不是模型已经完成的任务数量。RoboCasa 四任务的接口检查范围为 macOS／Apple Silicon、布局 11／风格 14／seed 0；其他场景组合及机械臂可达性需另行检查。修改文字指令不会改变上游评分器。
-策略只接收图像和机器人传感状态。**物体真值、仿真接触、奖励和任务成功信号不进入策略输入。** 独立评估可以读取仿真状态并终止 episode；模型自己声明 `done` 不等于任务成功。
+视觉策略接收图像与机器人传感状态；**Jev 只接收显式文字与本体状态**。物体真值、仿真接触、奖励和任务成功信号均不进入策略输入。独立评估可以读取仿真状态并终止 episode；模型声明 `done`、控制器返回 `reached`、指令序列结束，与任务成功分别记录。
-[查看真实手动调试界面：点动与夹爪控制、本体观测、执行反馈和独立评分](docs/images/workspace-feedback.png)。该图记录手动点动后的反馈，不是任务成功截图。
+本地工作台分为三个面板：
+
+| 面板 | 可以做什么 |
+| --- | --- |
+| 手动控制 | 沿固定轴或 TCP 自身轴平移、旋转，开合夹爪，提交六维组合增量，查看位置与姿态误差。 |
+| 模型运行 | 输入指令、选择连接、运行诊断，使用单步、连续执行、暂停与继续。 |
+| 场景设置 | 选择后端、任务、机器人和场景参数，点击应用后重建环境。 |
+
+“停止执行”跨面板保持可见。切换面板或画中画／双相机并排布局，不会启动推理或推进物理。当前任务、动作和请求结束后，才能提交手动控制。手动目标跟踪预算独立于模型预算；TCP 旋转不是单关节点动。详见[工作台使用说明](docs/WORKSPACE.md)和[六维手动控制](docs/MANUAL_CONTROL.md)。
## 第一步 · 打开本地工作台
@@ -59,7 +64,7 @@ py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m maniloop demo
```
-打开 [http://127.0.0.1:8765](http://127.0.0.1:8765)。选择机器人、布局和任务，应用配置，试用手动移动与夹爪控件，再重置场景。**这一步不需要 API Key。** 服务只监听本机地址；使用时保持终端运行，按 `Ctrl+C` 关闭。如需换端口，启动时加 `--port 8766`。
+打开 [http://127.0.0.1:8765](http://127.0.0.1:8765)。工作台默认显示“手动控制”；可到“场景设置”选择机器人、布局和任务，应用后返回手动面板试用机械臂与夹爪。**这一步不需要 API Key。** 服务只监听本机地址；使用时保持终端运行，按 `Ctrl+C` 关闭。如需换端口，启动时加 `--port 8766`。
继续以下步骤前，先停止服务，并保持在仓库根目录操作。**Windows 用户**将后续 `python` 命令替换为 `.\.venv\Scripts\python.exe`；多行 Bash 示例合并为一行，并去掉行尾的 `\`。安装 `uv` 后，运行任一安装脚本时加 `--uv .\.venv\Scripts\uv.exe`。无需执行激活脚本或修改执行策略。
@@ -67,27 +72,57 @@ py -3.12 -m venv .venv
-## 第二步 · 添加官方 LIBERO 任务
+## 第二步 · 选择仿真后端
+
+完成第一步即可使用内置场景。选择下方需要的可选后端，从仓库根目录运行对应命令；改用另一条启动示例前先停止现有 demo。各后端通过独立 Python worker 运行；启动工作台仍使用主环境的 `python`。
```bash
python -m pip install uv
+```
+
+### LIBERO
+
+```bash
python scripts/setup_libero.py
python -m maniloop list --backend libero --libero-suite libero_goal
python -m maniloop smoke --backend libero
python -m maniloop demo --backend libero --port 8767
```
-打开 [http://127.0.0.1:8767](http://127.0.0.1:8767)。在环境面板选择任务套件、任务 ID 与初始化 ID。ID 从零开始，`list` 会列出任务名称。默认是 `libero_spatial` 任务 0、初始化 0：将盘子与小碗之间的黑碗放到盘子上。
+打开 [http://127.0.0.1:8767](http://127.0.0.1:8767)。在“场景设置”选择任务套件、任务 ID 与初始化 ID。ID 从零开始，`list` 会列出任务名称。默认是 `libero_spatial` 任务 0、初始化 0：将盘子与小碗之间的黑碗放到盘子上。安装需要网络与数 GB 磁盘空间，无需示范数据集或策略权重。详见 [LIBERO 安装说明](docs/LIBERO.md)。
-安装器需要网络与数 GB 磁盘空间，无需示范数据集或策略权重。它下载固定版本的官方源码、资源与由项目管理的 Python，并将依赖分开：
+### robosuite
+
+```bash
+python scripts/setup_robosuite.py
+python -m maniloop list --backend robosuite
+python -m maniloop demo --backend robosuite --task Lift --port 8870
+```
+
+打开 [http://127.0.0.1:8870](http://127.0.0.1:8870)。五项 Panda 任务在独立环境使用 robosuite 1.5.2／MuJoCo 3.3.7。适配器明确采用世界系 OSC，并记录这一配置。相机、手动控制和模型接口已接入；这些任务尚无模型成功率结论。详见[控制语义与检查范围](docs/ROBOSUITE.md)。
+
+### RoboCasa
+
+```bash
+python scripts/setup_robocasa.py --download-assets
+python -m maniloop list --backend robocasa
+python -m maniloop demo --backend robocasa --task OpenDrawer \
+ --robocasa-layout 11 --robocasa-style 14 --port 8872
+```
+
+打开 [http://127.0.0.1:8872](http://127.0.0.1:8872)。安装器固定 RoboCasa 及其配套 robosuite 源码，与独立 robosuite 后端分开。五组必需资产解压约 11.1 GB，另需源码和运行环境空间。控制 PandaOmron 机械臂和夹爪时，底座接收零速度、躯干接收零增量；本接口不提供导航。默认场景接口检查不代表自主任务成功，也不保证所有场景都能仅靠机械臂完成。详见 [RoboCasa 安装与限制](docs/ROBOCASA.md)。
+
+### 保持运行环境独立
| 环境 | 职责 |
| --- | --- |
| `.venv` · Python 3.12 | 主程序、网页、模型连接与内置 MuJoCo 场景 |
| `.venv-libero` · Python 3.10 | LIBERO 仿真及固定依赖 |
+| `.venv-robosuite` · Python 3.11 | 独立 robosuite 仿真 |
+| `.venv-robocasa` · Python 3.11 | RoboCasa 仿真及兼容的上游依赖 |
| `.venv-vla` · Python 3.12 | 可选的 LeRobot 与学习策略推理 |
-三个环境保持隔离。详见 [LIBERO 安装与任务选择](docs/LIBERO.md)。
+不要将这些依赖合并安装到主环境。RoboCasa worker 是仿真专用安装，不是上游完整训练环境。可选资产、运行环境与模型权重在本地下载，不随仓库分发。
## 第三步 · 连接模型
@@ -98,12 +133,13 @@ python -m maniloop demo --backend libero --port 8767
| 通过 Codex 使用 GPT | 在官方 Codex CLI 登录，选择 Codex 来源 | 使用该 CLI／账户可用的模型；ManiLoop 不提取登录 token |
| 云端或自托管视觉模型 | 填写服务基础地址、模型 ID、API Key，或导入供应商 TOML | 机器人控制需要 **Responses API、图像输入和 JSON Schema 输出** |
| 本地 LeRobot 检查点 | 选择由独立推理进程加载的支持模型 | 当前提供三个已核对的 LIBERO 检查点预设，不是任意模型文件夹导入 |
+| Jev 文本原语 | 在模型面板填写独立的 TypeSafe 密钥 | 显式文字与机器人本体状态；有限的机械臂／夹爪指令，无图像与物体定位 |
查看实际本地程序中的模型配置
-
+
-*当前本地界面；这里的模型和任务设置用于真实执行。*
+*当前中文“模型运行”面板，载入的是内置 Panda 场景，尚未发起模型请求。这里的连接和指令用于真实执行，与网站使用已保存数据的体验工作台分开。*
### GPT 与兼容模型 API
@@ -115,7 +151,7 @@ codex login
python -m maniloop demo --backend libero --codex-login --port 8767
```
-选择 Codex 来源并加载 GPT 演示预设，从文字、图像、动作格式诊断或单个动作开始。安装了多个 CLI 版本时，可用 `MANILOOP_CODEX_BIN` 选择所需程序。详见 [GPT 操作说明](docs/GPT6_DEMO.md)。
+选择 Codex 来源并加载 GPT 演示预设。该预设会选择 LIBERO；在其他后端使用模型时，保留所选场景并直接配置连接。从文字、图像、动作格式诊断或单个动作开始。安装了多个 CLI 版本时，可用 `MANILOOP_CODEX_BIN` 选择所需程序。详见 [GPT 操作说明](docs/GPT6_DEMO.md)。
通过 API 连接时，选择手动填写或导入配置来源，填写模型 ID、供应商基础地址与密钥。可参考 [OpenAI 配置](examples/openai.example.toml)和[自定义供应商配置](examples/custom.example.toml)。自己部署的本地模型也可走这条路径，**前提是服务端实现所需的 Responses 协议**，例如使用本地地址 `http://127.0.0.1:8000/v1`。仅支持 Chat Completions 的服务不能用于机器人控制；独立的[文字聊天页](docs/API_CHAT.md)支持这两种协议。填写模型 ID 不代表模型一定支持视觉与结构化动作。模型请求使用所选账户额度或供应商计费；打开网页不会自动推理。
@@ -126,7 +162,7 @@ python scripts/setup_vla.py --models smolvla-libero act-libero diffusion-libero
python -m maniloop demo --backend libero --port 8767
```
-在页面选择本地 LeRobot 来源与已下载模型。省略 `--models` 则只安装 SmolVLA。推理可离线运行于 CPU、Apple MPS 或 CUDA，保留检查点保存的预处理与动作队列。
+在页面选择本地 LeRobot 来源与已下载模型。省略 `--models` 则只安装 SmolVLA。即使安装了其他后端，这些预设仍**仅用于 LIBERO**。推理可离线运行于 CPU、Apple MPS 或 CUDA，保留检查点保存的预处理与动作队列。
也可直接通过命令行调用本地检查点；将 `smolvla-libero` 替换为 `act-libero` 或 `diffusion-libero` 即可选择其他已安装的预设：
@@ -145,6 +181,20 @@ python -m maniloop benchmark --backend libero --agent lerobot \
如果这些预设**已经存放在另一块本地磁盘**，将 `MANILOOP_MODELS_ROOT` 设为它们的父目录。目录内须包含与预设同名的文件夹，如 `smolvla-libero/`，SmolVLA 还需要 `smolvlm-tokenizer/`。`MANILOOP_VLA_PYTHON` 可指向兼容的推理环境。这些设置用于更换受支持快照的存放位置，不会导入任意 Hugging Face 仓库或自定义权重。目录要求与固定版本见[本地模型指南](docs/VLA.md)。
+### Jev：显式文本指令
+
+在“模型运行”中选择 **Jev · 文本原语控制**，输入 TypeSafe API Key。该密钥与 OpenAI 连接分开，只保存在服务内存，不写入文件或浏览器存储。保存密钥不调用模型；连接诊断和指令执行会使用 TypeSafe API。无需安装 Jev 权重或 TypeSafe SDK。
+
+每行输入一个动作，例如：
+
+```text
+Move up along world Z by 10 mm.
+Rotate about the tool Z axis by 5 degrees.
+Open the gripper.
+```
+
+Jev 根据文字与机器人本体状态选择有限原语，单位、坐标系和执行由本地代码处理。它不接收相机图像，不能自主定位物体进行抓取。该模式要求受控时序；每组最多 20 行，遇到含糊指令、低候选概率、API 错误或控制失败会停止。现有仿真检查使用模拟 Jev 响应；真实服务能力需单独运行诊断或指令验证。详见 [Jev 设置与验证范围](docs/JEV.md)。
+
### 接入自己的检查点或策略
新的检查点需要单独注册模型，配置正确的预处理、动作映射与来源信息。不要直接替换预设文件后仍将它标作原预设。若要使用自己的推理实现，可实现 [`Agent`](src/maniloop/agents/base.py) 的 `reset()` 与 `decide(task, observation, images, history, geometry_results)`，返回动作字典或 [`ActionChunk`](src/maniloop/core/actions.py)，再接入 `EpisodeRunner`，按需增加 CLI／UI 选项；当前没有自动加载任意检查点的通用入口。
@@ -153,7 +203,7 @@ python -m maniloop benchmark --backend libero --agent lerobot \
## 第四步 · 录制完整尝试，检查结果
-以下命令从官方初始化运行一次新的 GPT-6 尝试，并录制每个 LIBERO 控制步：
+普通模型运行可查看决策日志、控制反馈、前后帧和独立结果。完整逐控制步录制与视频导出目前要求 **LIBERO 和受控时序**。以下命令从官方初始化运行一次新的 GPT-6 尝试，并录制每个控制步：
```bash
python -m maniloop benchmark \
@@ -191,7 +241,7 @@ python -m maniloop.recording.video runs/gpt-demo/EPISODE_ID/recording
## 使用指南、检查与适用范围
-[LIBERO 任务](docs/LIBERO.md) · [GPT 操作](docs/GPT6_DEMO.md) · [本地检查点](docs/VLA.md) · [API 文字聊天](docs/API_CHAT.md) · [录制](docs/EPISODE_RECORDING.md) · [架构](docs/ARCHITECTURE.md)
+[工作台](docs/WORKSPACE.md) · [手动控制](docs/MANUAL_CONTROL.md) · [LIBERO](docs/LIBERO.md) · [robosuite](docs/ROBOSUITE.md) · [RoboCasa](docs/ROBOCASA.md) · [GPT 操作](docs/GPT6_DEMO.md) · [Jev](docs/JEV.md) · [本地检查点](docs/VLA.md) · [API 文字聊天](docs/API_CHAT.md) · [录制](docs/EPISODE_RECORDING.md) · [架构](docs/ARCHITECTURE.md)
```bash
python -m pip install -r requirements-dev.txt
@@ -199,53 +249,10 @@ python -m pytest -q
python -m maniloop smoke
```
-这些检查不发送模型请求。离线测试覆盖 macOS、Linux 和 Windows；渲染已在 macOS 与 Linux OSMesa 上验证，Windows 渲染和 LIBERO 运行尚未验证。渲染需要图形后端。ManiLoop 用于仿真：ARX 夹爪采用近似模型，软件未作为真机控制或安全系统验收。使用官方 LIBERO 资源，不等于具有论文级基准可比性。
+这些检查不发送模型请求。离线测试覆盖 macOS、Linux 和 Windows；主程序渲染已在 macOS 与 Linux OSMesa 上验证。可选后端的真实集成检查需按各自指南显式启用；普通 pytest 不会验证尚未安装的后端。Windows 上的渲染与 LIBERO 运行尚未验证。渲染需要图形后端。ManiLoop 用于仿真：ARX 夹爪采用近似模型，软件未作为真机控制或安全系统验收。环境接入、控制器检查与真实模型任务结果分别记录。
## 许可证
-ManiLoop 采用 [MIT 许可证](LICENSE)。ARX X5 资产保留 MIT，Panda 资产保留 Apache-2.0。LIBERO、LeRobot 和下载的检查点分别保留自己的许可证。详见[第三方声明](THIRD_PARTY_NOTICES.md)。
-
-## 可选 robosuite 操作任务
-
-使用独立环境接入 Panda 的 Lift、Stack、PickPlaceCan、Door、NutAssemblySquare，
-不改变已有 LIBERO。复用当前工作台的手动控制、模型入口与决策回看。
-安装、控制协议和限制见 [robosuite 指南](docs/ROBOSUITE.md)。
-
-```bash
-python scripts/setup_robosuite.py
-python -m maniloop demo --backend robosuite --task Lift --port 8870
-```
-
-任务已接入不代表模型已完成任务。现有 LIBERO 权重仍只用于 LIBERO；此后端尚不包含 RoboCasa。
-
-## 可选 RoboCasa 厨房任务
-
-新增 `OpenDrawer`、`CloseDrawer`、`OpenCabinet`、`CoffeeSetupMug` 四项厨房任务，
-使用独立的 PandaOmron 后端。首版控制机械臂和夹爪，底座速度与躯干增量保持0，不包含导航。
-
-```bash
-python scripts/setup_robocasa.py --download-assets
-python -m maniloop demo --backend robocasa --task OpenDrawer --port 8872
-```
-
-继续使用同一个工作台查看图像、手动控制、接入模型与回看决策。安装器固定两个上游源码版本，
-将依赖隔离到新环境，不改变现有 robosuite / LIBERO。五组必需资产解压约11.1 GB，另需源码和运行环境空间。
-四任务已在默认布局11/风格14/种子0完成真实接口检查；这不代表模型成功，也不保证其他场景
-在底座保持时一定可完成。详细说明见 [RoboCasa 安装与控制指南](docs/ROBOCASA.md)。
-
-
-### 工作台分区
-
-手动控制、模型运行和场景设置分区显示，相机与动作按钮并排。切换分区保留当前表单；相机布局切换不推进物理。参见[工作台使用说明](docs/WORKSPACE.md)和[六维手动控制](docs/MANUAL_CONTROL.md)。
-
-### Jev 文本指令
-
-在模型面板选择“Jev · 文本原语控制”，将每行一个明确指令映射为有界的机械臂或夹爪动作。
-TypeSafe 专用密钥可直接在工作台填写，只保存在服务内存，与 OpenAI 连接分开。
-保存密钥不调用模型；连接诊断和执行会调用 TypeSafe API。
-
-Jev 只接收文字和机器人本体状态，不读取相机或物体状态。当前支持移动10毫米、
-绕工具轴旋转5度等原语，不提供自主视觉定位或抓放；指令序列结束也不代表任务成功。
-安装、示例和验证范围见 [Jev 使用说明](docs/JEV.md)。
+ManiLoop 采用 [MIT 许可证](LICENSE)。ARX X5 资产保留 MIT，Panda 资产保留 Apache-2.0。LIBERO、robosuite、RoboCasa、LeRobot 及下载的资产和检查点分别保留自己的许可证。详见[第三方声明](THIRD_PARTY_NOTICES.md)与[贡献指南](CONTRIBUTING.md)。
diff --git a/docs/images/current-workspace-manual.png b/docs/images/current-workspace-manual.png
new file mode 100644
index 0000000..3685c6b
Binary files /dev/null and b/docs/images/current-workspace-manual.png differ
diff --git a/docs/images/current-workspace-model.png b/docs/images/current-workspace-model.png
new file mode 100644
index 0000000..b0dac3e
Binary files /dev/null and b/docs/images/current-workspace-model.png differ
diff --git a/site/README.md b/site/README.md
index 7a6887d..48b8f8e 100644
--- a/site/README.md
+++ b/site/README.md
@@ -2,7 +2,7 @@
[English](https://github.com/Daniel-rmc/ManiLoop/blob/main/README.md) · [中文](https://github.com/Daniel-rmc/ManiLoop/blob/main/README.zh-CN.md) · [Project website](https://daniel-rmc.github.io/ManiLoop/)
-The site introduces the implemented simulation framework, shows a complete GPT-controlled LIBERO episode, compares recorded model examples, and provides setup instructions. The [interactive workspace](https://daniel-rmc.github.io/ManiLoop/playground/?lang=en) lets visitors inspect the actual recorded observations, actions and feedback. Live simulation and inference run in the local application.
+The site introduces the implemented simulation framework across the built-in MuJoCo, LIBERO, robosuite and RoboCasa backends. It presents the modular local workspace, six-dimensional manual TCP control and GPT, local VLA and Jev policy connections, alongside complete recorded LIBERO episodes and setup instructions. The [interactive workspace](https://daniel-rmc.github.io/ManiLoop/playground/?lang=en) lets visitors inspect the actual recorded observations, actions and feedback. Live simulation and inference run in the local application.
English and Chinese are available throughout the site. `?lang=en` or `?lang=zh` selects a language explicitly; otherwise the saved preference is used, followed by the browser language. The language control updates the URL and saves the choice under `maniloop-language`. The workspace uses the same preference. There are no external fonts, analytics, credentials or model requests on this site.
@@ -16,6 +16,16 @@ python3 -m http.server 8080 --bind 127.0.0.1 --directory site
Open . The static assets use relative paths and also work under `/ManiLoop/`; no package installation or build step is required.
+## Current local workspace and backend overview
+
+The framework section distinguishes four environment adapters from the selected LIBERO episode results. Task availability and integration checks are not model success claims. Existing LeRobot presets remain LIBERO-specific. robosuite and RoboCasa provide decision-level observations and replay, but not complete control-step video export.
+
+`assets/workspace/current-workspace-manual.png` and `current-workspace-model.png` are real screenshots of the current local application using the built-in Panda tabletop scene. They show the separate manual and model panels, retain their original aspect ratios, and open at full size. They are static presentation assets; the website does not execute their controls. The local application also has a separate scene-settings panel and a persistent stop control. The interactive `playground/` remains a viewer for a recorded LIBERO episode.
+
+The Jev description refers to text primitive selection from explicit instructions and public proprioception. It uses a separate TypeSafe API key, receives no images, and delegates unit conversion and motion execution to local code. It is not presented as visual localization or autonomous grasping, and is not added to the recorded model-result table.
+
+New presentation copy uses the existing `app.js` English/Chinese dictionaries and `data-i18n` attributes. `showcase.js`, the three recorded skill videos, their JSON manifests, and the playground remain independent of this presentation update.
+
## Episode and model results
The homepage plays a muted MP4 preview with synchronized, saved model action notes. Play/pause, previous/next decision and the time slider let visitors inspect the requested command and the **preceding** action's feedback. The preview uses simulation time; model waiting time is omitted. It is recorded output, not live inference. Reduced-motion preferences start with a still poster, and a failed autoplay attempt keeps a manual play control. Off-screen video pauses; a visitor's explicit pause is preserved.
@@ -42,21 +52,26 @@ Published camera images and videos come from simulator rendering. ManiLoop sourc
## Publish
-Source files live in `main:site/`; GitHub Pages serves the root of `gh-pages`. After committing a reviewed site update:
+Source files live in `main:site/`; GitHub Pages serves the root of `gh-pages`. Make site changes on a feature branch and open a pull request for review before merging into `main`.
+
+After the reviewed pull request is merged, publish `site/` from an up-to-date `main` checkout using the repository's authorized Pages publishing process. The existing branch publication command is:
```bash
-git push origin main
git subtree push --prefix site origin gh-pages
```
-Use a normal fast-forward push. `.nojekyll` keeps this a static site.
+Use a normal fast-forward push. `.nojekyll` keeps this a static site. A reviewed site-only update can also be published from its feature branch while the source pull request is pending; the source commit and Pages commit should be recorded in the pull request.
## 中文说明
-官网展示已实现的框架功能、完整 GPT 仿真操作录像、各模型的实际运行示例与安装方法。[交互工作台](https://daniel-rmc.github.io/ManiLoop/playground/?lang=zh)可逐步查看真实运行的观察、动作与反馈；实时仿真和模型推理在本地应用中运行。
+官网展示内置 MuJoCo、LIBERO、robosuite、RoboCasa 四类后端、模块化本地工作台、六维手动位姿控制，以及 GPT、本地 VLA 与 Jev 文本原语入口；保留完整 GPT 仿真操作录像、实际运行示例和安装方法。[交互工作台](https://daniel-rmc.github.io/ManiLoop/playground/?lang=zh)可逐步查看真实运行的观察、动作与反馈；实时仿真和模型推理在本地应用中运行。
全站支持英文与中文。URL 参数 `lang` 优先，其次为已保存的语言偏好，最后采用浏览器语言。切换语言会同步入口链接与偏好，不会重播正在播放的视频。页面不收集密钥，也不发起模型请求。
首页短片按仿真时间静音循环，同步展示模型保存的动作说明、请求动作和上一步执行反馈；未展示模型等待时间。可暂停、前后切换决策或拖动进度，不是实时推理。系统选择减少动态效果时，默认展示静止画面；自动播放失败时可手动播放。任务卡来自 `assets/showcase.json`，只接收官方独立评分已经验证成功的回合。所有卡片明确说明是单次回合最终结果。
模型结果为单次示例，不代表基准成功率。配置、动作块、预算与计时口径不同，不能由步数或耗时进行速度排名。完整运行条件见[实际结果](https://github.com/Daniel-rmc/ManiLoop/blob/main/docs/RESULTS.md)，本地安装见[中文使用说明](https://github.com/Daniel-rmc/ManiLoop/blob/main/README.zh-CN.md)。
+
+当前本地工作台截图使用内置 Panda 桌面场景，展示手动控制与模型运行分区，按原始比例显示，可点击查看大图。它们不是网页内嵌仿真；`playground/` 仍用于检查已录制的 LIBERO 回合。新后端的任务目录与接入检查不等于模型成功，既有 LeRobot 预设仍限 LIBERO。Jev 是独立 TypeSafe 认证的文本原语模式，不输入图像，不声称具备视觉定位或自主抓取能力。
+
+主页变更在功能分支完成，通过 PR 合入受保护的 `main`，然后再按授权的 Pages 发布流程同步 `site/`；不直接向 `main` 推送。
diff --git a/site/app.js b/site/app.js
index 6958243..142d1ae 100644
--- a/site/app.js
+++ b/site/app.js
@@ -15,30 +15,30 @@
"hero_see": "See.",
"hero_act": "Act.",
"hero_observe": "Observe again.",
- "hero_description": "Connect model decisions to a physical feedback loop. Run GPT and local VLA policies in MuJoCo, inspect each action, and replay the complete episode.",
- "try_workspace": "Try the workspace",
+ "hero_description": "Connect model decisions to a physical feedback loop. Explore MuJoCo, LIBERO, robosuite and RoboCasa through one local workspace, with explicit actions and inspectable feedback.",
+ "try_workspace": "Explore the recorded workspace",
"watch_demo": "Watch the demo",
- "workspace_hint": "Explore the recorded run interactively in your browser. Live simulation runs locally.",
+ "workspace_hint": "Inspect a saved LIBERO run in your browser. The live, modular workspace runs on your machine.",
"tech_stack": "Technology stack",
"observation_label": "OBSERVATION / 01",
"hero_alt": "A recorded LIBERO scene with the Panda arm, bowls and a plate",
"external_camera": "EXTERNAL CAMERA",
"real_image": "An actual simulator frame",
"browse_episode": "Explore the episode",
- "sensor_note": "The policy sees images and robot state.",
+ "sensor_note": "The policy receives only declared sensor observations.",
"score_note": "The evaluator stays separate.",
"framework_eyebrow": "01 / THE FRAMEWORK",
"framework_title": "Close the loop. Inspect every step.",
- "framework_intro": "A shared interface for model inference, simulator control and recorded feedback. See what the policy asked for and what the robot actually did.",
- "loop_aria": "Cameras and robot state feed the policy. Validated actions execute in the simulator, then fresh sensor feedback informs the next decision. Evaluation is separate.",
+ "framework_intro": "Connect environment adapters, model policies and local control. Inspect the requested action, the measured motion and the independently recorded result.",
+ "loop_aria": "Declared sensor observations feed the selected policy. Validated actions execute in a MuJoCo backend, then fresh sensor feedback informs the next decision. Evaluation is separate.",
"observe": "Observe",
- "observe_inputs": "Dual-camera RGB\nRobot proprioception",
+ "observe_inputs": "Camera RGB where supported\nRobot proprioception",
"observe_tag": "SENSOR INPUT",
"decide": "Decide",
- "decide_inputs": "Structured GPT actions\nor a local VLA policy",
+ "decide_inputs": "GPT / VLA policies\nJev text primitives",
"decide_tag": "POLICY",
"act": "Act",
- "act_inputs": "Validate and track targets\nMuJoCo / LIBERO",
+ "act_inputs": "Validate and track targets\nMuJoCo backends",
"act_tag": "LOCAL CONTROL",
"check": "Check",
"check_inputs": "New images and motion\nExecution status and error",
@@ -49,15 +49,15 @@
"boundary_label": "EVALUATOR ≠ POLICY INPUT",
"feature_a_tag": "A / INTERFACES",
"feature_a_title": "One framework, multiple policies",
- "feature_a_description": "Use GPT vision and structured output, or run SmolVLA, Diffusion Policy and ACT in an isolated inference environment. Each adapter preserves its observation and action protocol.",
+ "feature_a_description": "Connect GPT vision and structured actions, LIBERO-specific local VLA presets, or Jev text primitives. Each policy keeps its declared inputs and action protocol.",
"architecture_link": "Explore the architecture",
"feature_b_tag": "B / CONTROL",
- "feature_b_title": "Debug a single action",
- "feature_b_description": "Step once, pause after an action, or continue. Inspect the requested TCP target, actual displacement and controller error before allowing the next decision.",
- "control_link": "Use the control interface",
+ "feature_b_title": "Control a 6D TCP target",
+ "feature_b_description": "Jog XYZ, rotate Rx/Ry/Rz, or combine translation and rotation in fixed or tool axes. Inspect position and orientation error. Manual control needs no model key.",
+ "control_link": "Understand manual pose control",
"feature_c_tag": "C / RECORDING",
- "feature_c_title": "Replay the complete episode",
- "feature_c_description": "Record the initial scene and every native control step. Review both cameras, action notes and the independent result, or export a continuous MP4.",
+ "feature_c_title": "Keep actions and evidence together",
+ "feature_c_description": "Inspect decision-level observations, action notes and feedback across backends. LIBERO additionally supports complete control-step recordings and continuous MP4 export.",
"recording_link": "Record and export",
"demo_eyebrow": "02 / THE EPISODE",
"demo_title": "One task. Every step recorded.",
@@ -115,11 +115,11 @@
"comparison_results": "Check configurations and results",
"start_eyebrow": "04 / GET STARTED",
"start_title": "Run the loop on your machine.",
- "start_intro": "Start with simulation and manual control, then connect a model. The main application, LIBERO and VLA inference use separate environments.",
+ "start_intro": "Start with the local workspace and manual control, then choose an optional backend and policy. Simulation runtimes and VLA inference stay isolated.",
"install_main_title": "Set up the main environment",
"copy_main": "Copy main environment commands",
"copy": "Copy",
- "install_main_note": "Open 127.0.0.1:8765 for the default tabletop demo. Starting the simulator and using manual controls requires no model key. Use another terminal, or stop the demo with Ctrl+C, before installing LIBERO.",
+ "install_main_note": "Open 127.0.0.1:8765 for the built-in tabletop workspace. Manual controls need no model key. Use another terminal, or stop with Ctrl+C, before installing an optional backend.",
"install_full_link": "Full setup and Windows instructions",
"install_libero_title": "Add LIBERO",
"install_libero_subtitle": "A separate Python 3.10 simulation environment",
@@ -128,7 +128,7 @@
"libero_requirements": "Requirements and headless setup",
"connect_eyebrow": "CONNECT A MODEL",
"connect_title": "Give the policy a turn.",
- "connect_description": "Connect an API service with image and structured-output support, or use the ChatGPT login channel of a recent official Codex CLI. Run diagnostics, then try a small movement. Model requests use your provider or account quota.",
+ "connect_description": "Choose GPT through a compatible image and structured-output API or official Codex login, a local LIBERO VLA preset, or Jev text primitives with a separate TypeSafe API key. Diagnose the connection before a small action.",
"gpt_guide": "GPT simulation guide",
"api_guide": "API chat and diagnostics",
"vla_guide": "Local LeRobot / VLA policies",
@@ -140,7 +140,7 @@
"documentation": "Documentation",
"footer_note": "Recorded episodes here. Simulation runs locally.",
"page_title": "ManiLoop — See. Act. Observe again.",
- "meta_description": "Connect GPT and local VLA policies to MuJoCo and LIBERO. Inspect visual feedback, structured actions, complete episodes and independent task results.",
+ "meta_description": "Explore four MuJoCo simulation backends with GPT, local VLA policies, Jev text primitives and a modular workspace for 6D control, recorded feedback and independent evaluation.",
"og_description": "An open robot manipulation framework with visual feedback, local control and independently scored episodes.",
"missing_model": "Model not recorded",
"missing_date": "Date not recorded",
@@ -207,7 +207,40 @@
"comparison_unavailable": "Model examples could not be loaded. Read the result report below.",
"comparison_sim_time": "Simulation: {seconds} s",
"comparison_wall_time": "Wall time: {seconds} s",
- "comparison_wall_time_unknown": "Wall time: not recorded"
+ "comparison_wall_time_unknown": "Wall time: not recorded",
+ "environments_title": "Four environments. Explicit boundaries.",
+ "environments_intro": "Choose the scene in the local workspace. Optional backends keep separate runtimes, controllers and independent task evaluators.",
+ "env_builtin": "Built-in MuJoCo",
+ "env_builtin_description": "Panda and ARX5 tabletop scenes for manual control, pick-and-place and pushing experiments.",
+ "env_libero_description": "Official task suites and initializations. The recorded GPT examples and local VLA presets on this page use LIBERO.",
+ "env_robosuite_description": "Five Panda tasks: Lift, Stack, PickPlaceCan, Door and NutAssemblySquare, with dual-camera observation and local control.",
+ "env_robocasa_description": "Four kitchen tasks: open or close a drawer, open a cabinet, and position a mug. Arm and gripper control; no navigation planner.",
+ "env_documentation": "Setup and scope",
+ "environments_scope": "Backend integration is not a model success result. The existing LeRobot presets remain LIBERO-specific; robosuite and RoboCasa do not yet provide complete control-step video export.",
+ "workspace_eyebrow": "THE LOCAL APPLICATION",
+ "workspace_title": "A workspace for the whole loop.",
+ "workspace_description": "Manual control, model runs and scene settings have dedicated panels beside the cameras. These actual local-application screenshots use the built-in Panda tabletop scene; the website does not run their controls.",
+ "workspace_manual_label": "01 / MANUAL CONTROL",
+ "workspace_manual_alt": "The actual ManiLoop local workspace with camera observations and six-dimensional manual pose controls",
+ "workspace_manual_caption": "Fixed or tool axes, translation, rotation and gripper control. Position and orientation feedback stay visible.",
+ "workspace_model_label": "02 / MODEL RUNS",
+ "workspace_model_alt": "The actual ManiLoop local workspace showing its separate model connection and task-running panel",
+ "workspace_model_caption": "Connect a policy, inspect its settings, then step or run. Opening a panel does not submit a model request.",
+ "workspace_full_image": "Open full-size screenshot",
+ "workspace_scene_title": "Apply a scene when you are ready",
+ "workspace_scene_description": "Choose a backend and task, then apply to rebuild the environment. Editing a task instruction does not change the official evaluator. The stop control remains available across panels.",
+ "workspace_guide": "Local workspace guide",
+ "optional_backends_title": "Add another environment",
+ "optional_backends_description": "Use the dedicated installers for robosuite and RoboCasa. Each creates its own Python 3.11 runtime; RoboCasa additionally needs kitchen assets.",
+ "robosuite_setup": "Set up robosuite",
+ "robocasa_setup": "Set up RoboCasa",
+ "jev_guide": "Jev text primitives",
+ "jev_scope": "Jev receives explicit instructions and public proprioception, not images. It selects a bounded primitive; local code handles units, coordinates and motion. It does not visually locate objects or provide autonomous grasping.",
+ "model_requests_note": "Model requests use the selected provider or account quota. Switching workspace panels does not trigger inference.",
+ "env_builtin_tag": "01 / TABLETOP",
+ "env_libero_tag": "02 / BENCHMARK",
+ "env_robosuite_tag": "03 / MANIPULATION",
+ "env_robocasa_tag": "04 / KITCHEN"
},
"zh": {
"skip": "跳至正文",
@@ -222,30 +255,30 @@
"hero_see": "让模型看见、",
"hero_act": "行动，",
"hero_observe": "再观察。",
- "hero_description": "把模型判断接入物理反馈闭环。在 MuJoCo 中运行 GPT 和本地 VLA 策略，检查每次动作，并回看完整操作过程。",
+ "hero_description": "把模型判断放进物理闭环。在同一个本地工作台中连接 MuJoCo、LIBERO、robosuite 与 RoboCasa，检查明确的动作和实际执行反馈。",
"watch_demo": "观看完整演示",
- "try_workspace": "体验工作台",
- "workspace_hint": "在浏览器中交互查看实际运行记录；实时仿真在本地启动。",
+ "try_workspace": "探索回放工作台",
+ "workspace_hint": "在浏览器中逐步检查已保存的 LIBERO 回合。实时的模块化工作台在你的电脑上运行。",
"tech_stack": "技术栈",
"observation_label": "观察画面 / 01",
"hero_alt": "真实 LIBERO 画面：Panda 机械臂、碗与盘子",
"external_camera": "外部相机",
"real_image": "仿真器实际渲染画面",
"browse_episode": "查看完整过程",
- "sensor_note": "策略读取图像与机器人本体状态。",
+ "sensor_note": "策略只接收显式提供的传感观测。",
"score_note": "独立评分与策略输入隔离。",
"framework_eyebrow": "01 / 框架",
"framework_title": "形成闭环，看清每一步。",
- "framework_intro": "用统一接口连接模型推理、仿真控制和执行记录。对照策略请求与机器人实际动作，定位发生了什么。",
- "loop_aria": "相机与机器人本体状态输入策略，经过校验的动作在仿真中执行，再将新的传感反馈用于下一次判断。独立评分与策略输入隔离。",
+ "framework_intro": "连接环境适配器、模型策略与本地控制。核对请求动作、实测运动和独立记录的任务结果。",
+ "loop_aria": "显式传感观测进入所选策略，校验后的动作在 MuJoCo 后端执行，新的传感反馈再用于下一次决策。任务评分独立保留。",
"observe": "观察",
- "observe_inputs": "双相机 RGB\n机器人本体状态",
+ "observe_inputs": "按策略支持提供相机 RGB\n机器人本体状态",
"observe_tag": "传感器输入",
"decide": "判断",
- "decide_inputs": "GPT 结构化动作\n或本地 VLA 策略",
+ "decide_inputs": "GPT / VLA 策略\nJev 文本原语",
"decide_tag": "策略推理",
"act": "执行",
- "act_inputs": "动作校验与目标跟踪\nMuJoCo / LIBERO",
+ "act_inputs": "动作校验与目标跟踪\nMuJoCo 仿真后端",
"act_tag": "本地控制",
"check": "核对",
"check_inputs": "新图像与实际位移\n执行状态与控制误差",
@@ -256,15 +289,15 @@
"boundary_label": "独立评分 ≠ 策略输入",
"feature_a_tag": "A / 策略接口",
"feature_a_title": "同一框架，多种策略",
- "feature_a_description": "连接 GPT 视觉与结构化输出，或在隔离的推理环境中运行 SmolVLA、Diffusion Policy 和 ACT。各适配器保留模型对应的观测与动作协议。",
+ "feature_a_description": "连接 GPT 视觉与结构化动作、LIBERO 专用本地 VLA 预设，或 Jev 文本原语。每种策略保留其声明的输入与动作协议。",
"architecture_link": "查看架构",
"feature_b_tag": "B / 动作控制",
- "feature_b_title": "从一个动作开始调试",
- "feature_b_description": "单步执行、动作后暂停或连续运行。在下一次判断前，查看请求的 TCP 目标、实际位移和控制误差。",
- "control_link": "使用控制界面",
+ "feature_b_title": "手动控制六维末端目标",
+ "feature_b_description": "在固定轴或末端自身轴下点动 XYZ、旋转 Rx/Ry/Rz，也可组合六维增量。查看位置与姿态误差；手动控制无需模型密钥。",
+ "control_link": "阅读手动位姿控制说明",
"feature_c_tag": "C / 过程记录",
- "feature_c_title": "回看完整 episode",
- "feature_c_description": "记录初始场景与每个原生控制步。回看双相机画面、动作说明和独立结果，或导出连续 MP4 录像。",
+ "feature_c_title": "把动作与证据一起保留",
+ "feature_c_description": "跨后端检查决策级观测、动作说明和反馈。LIBERO 还支持记录每个原生控制步，并导出连续 MP4 录像。",
"recording_link": "录制与导出",
"demo_eyebrow": "02 / 实际演示",
"demo_title": "一项任务，记录完整过程。",
@@ -322,11 +355,11 @@
"comparison_results": "核对模型配置与结果",
"start_eyebrow": "04 / 开始使用",
"start_title": "在你的电脑上运行闭环。",
- "start_intro": "先启动仿真与手动控制，再接入模型。主程序、LIBERO 与 VLA 推理分别使用独立环境。",
+ "start_intro": "先启动本地工作台和手动控制，再选择可选后端与策略。各仿真运行环境与 VLA 推理保持隔离。",
"install_main_title": "准备主环境",
"copy_main": "复制主环境安装命令",
"copy": "复制",
- "install_main_note": "在本机打开 127.0.0.1:8765 进入默认桌面演示。启动仿真和手动控制不需要模型密钥。安装 LIBERO 时另开终端，或先按 Ctrl+C 停止演示。",
+ "install_main_note": "在本机打开 127.0.0.1:8765，进入内置桌面工作台。手动控制无需模型密钥。安装可选后端前，另开终端或先按 Ctrl+C 停止服务。",
"install_full_link": "完整安装与 Windows 说明",
"install_libero_title": "安装 LIBERO",
"install_libero_subtitle": "独立的 Python 3.10 仿真环境",
@@ -335,7 +368,7 @@
"libero_requirements": "环境要求与无显示器配置",
"connect_eyebrow": "接入模型",
"connect_title": "让策略开始操作。",
- "connect_description": "使用支持图像和结构化输出的 API 服务，或近期官方 Codex CLI 的 ChatGPT 登录通道。先运行诊断，再尝试一个小位移。模型请求使用相应服务或账户额度。",
+ "connect_description": "选择兼容图像和结构化输出的 GPT API 或官方 Codex 登录、本地 LIBERO VLA 预设，或使用独立 TypeSafe API Key 的 Jev 文本原语。先检查连接，再测试小动作。",
"gpt_guide": "GPT 仿真操作指南",
"api_guide": "API 聊天与诊断",
"vla_guide": "本地 LeRobot / VLA 策略",
@@ -347,7 +380,7 @@
"documentation": "使用文档",
"footer_note": "此页回放已录制过程，仿真在本地运行。",
"page_title": "ManiLoop — 让模型看见、行动，再观察。",
- "meta_description": "连接 GPT、本地 VLA 与 MuJoCo / LIBERO，检查视觉反馈和结构化动作，回看完整 episode 与独立任务结果。",
+ "meta_description": "在四类 MuJoCo 仿真后端中连接 GPT、本地 VLA 与 Jev 文本原语，使用模块化工作台进行六维控制、反馈回看与独立评分。",
"og_description": "具备视觉反馈、本地控制与独立评分 episode 的开放机器人操作框架。",
"missing_model": "模型未记录",
"missing_date": "日期未记录",
@@ -414,7 +447,40 @@
"comparison_unavailable": "模型示例暂时无法载入，请阅读下方结果文档。",
"comparison_sim_time": "仿真时间：{seconds} 秒",
"comparison_wall_time": "墙钟时间：{seconds} 秒",
- "comparison_wall_time_unknown": "墙钟时间：未记录"
+ "comparison_wall_time_unknown": "墙钟时间：未记录",
+ "environments_title": "四类环境，明确各自的范围。",
+ "environments_intro": "在本地工作台选择场景。可选后端分别维护运行环境、控制器与独立任务评分。",
+ "env_builtin": "内置 MuJoCo",
+ "env_builtin_description": "Panda 与 ARX5 桌面场景，用于手动控制、抓放和推动实验。",
+ "env_libero_description": "沿用官方任务集与初始化。本页的 GPT 操作录像和本地 VLA 预设使用 LIBERO。",
+ "env_robosuite_description": "接入 Panda 的 Lift、Stack、PickPlaceCan、Door 与 NutAssemblySquare 五项任务，提供双相机和本地控制。",
+ "env_robocasa_description": "四项厨房任务：开抽屉、关抽屉、开柜门和摆放杯子。控制机械臂与夹爪，不包含导航规划。",
+ "env_documentation": "安装与范围",
+ "environments_scope": "后端接入不代表模型已完成任务。既有 LeRobot 预设仍限 LIBERO；robosuite 与 RoboCasa 暂不提供完整逐控制步视频导出。",
+ "workspace_eyebrow": "本地应用",
+ "workspace_title": "为完整闭环组织的工作台。",
+ "workspace_description": "手动控制、模型运行与场景设置各有独立分区，与相机画面并排。下方为内置 Panda 桌面场景的真实本地工作台截图，网站不会执行图中的控件。",
+ "workspace_manual_label": "01 / 手动控制",
+ "workspace_manual_alt": "ManiLoop 当前本地工作台：相机观测与六维手动位姿控制面板",
+ "workspace_manual_caption": "固定轴或末端自身轴、平移、旋转和夹爪控制，配合可直接查看的位置与姿态反馈。",
+ "workspace_model_label": "02 / 模型运行",
+ "workspace_model_alt": "ManiLoop 当前本地工作台：独立的模型连接与任务运行面板",
+ "workspace_model_caption": "连接策略、核对参数，再单步或连续执行。切换到模型分区不会发起模型请求。",
+ "workspace_full_image": "查看原始尺寸截图",
+ "workspace_scene_title": "确认后再应用场景",
+ "workspace_scene_description": "选择后端与任务，点击应用才会重建环境。修改任务指令不改变官方评分规则；停止按钮在各分区始终可用。",
+ "workspace_guide": "本地工作台指南",
+ "optional_backends_title": "接入其他环境",
+ "optional_backends_description": "robosuite 和 RoboCasa 使用各自的安装器，分别创建独立 Python 3.11 运行环境；RoboCasa 还需下载厨房资产。",
+ "robosuite_setup": "安装 robosuite",
+ "robocasa_setup": "安装 RoboCasa",
+ "jev_guide": "Jev 文本原语控制",
+ "jev_scope": "Jev 接收明确指令与公开本体状态，不接收图像。它选择有界动作原语，由本地代码处理单位、坐标与运动；不进行视觉定位或自主抓取。",
+ "model_requests_note": "模型请求使用所选服务或账户额度。切换工作台分区不会触发推理。",
+ "env_builtin_tag": "01 / 桌面",
+ "env_libero_tag": "02 / 基准任务",
+ "env_robosuite_tag": "03 / 操作任务",
+ "env_robocasa_tag": "04 / 厨房"
}
};
const byId = (id) => document.getElementById(id);
diff --git a/site/assets/workspace/current-workspace-manual.png b/site/assets/workspace/current-workspace-manual.png
new file mode 100644
index 0000000..3685c6b
Binary files /dev/null and b/site/assets/workspace/current-workspace-manual.png differ
diff --git a/site/assets/workspace/current-workspace-model.png b/site/assets/workspace/current-workspace-model.png
new file mode 100644
index 0000000..b0dac3e
Binary files /dev/null and b/site/assets/workspace/current-workspace-model.png differ
diff --git a/site/index.html b/site/index.html
index 88e4edf..238d9fd 100644
--- a/site/index.html
+++ b/site/index.html
@@ -4,15 +4,15 @@
-
+
ManiLoop — See. Act. Observe again.
-
-
+
+
@@ -27,9 +27,9 @@
OPEN ROBOT MANIPULATION
See. Act. Observe again.
-
Connect model decisions to a physical feedback loop. Run GPT and local VLA policies in MuJoCo, inspect each action, and replay the complete episode.
Explore the recorded run interactively in your browser. Live simulation runs locally.
-
Python 3.12MuJoCoLIBEROGPT + VLA
+
Connect model decisions to a physical feedback loop. Explore MuJoCo, LIBERO, robosuite and RoboCasa through one local workspace, with explicit actions and inspectable feedback.
Inspect a saved LIBERO run in your browser. The live, modular workspace runs on your machine.
+
Python 3.12MuJoCoLIBEROrobosuiteRoboCasa
RECORDED POLICY LOOPGPT · LIBERO · PANDA
@@ -48,7 +48,7 @@
See. Feedback to the model uses only camera images and proprioception available on a real robot; no privileged simulator ground truth is provided. The controller computes TCP error from the model's previous command and the measured post-action TCP position, without object ground truth. Action execution completion does not mean task success.
Pause to read the model's saved action note. This page makes no model requests.
-
∥
The policy sees images and robot state. The evaluator stays separate.
+
∥
The policy receives only declared sensor observations. The evaluator stays separate.
@@ -57,15 +57,15 @@
See.
Run time includes model planning and waiting, simulation execution and recording; environment initialization and policy loading are excluded. Video playback follows simulation time.
A shared interface for model inference, simulator control and recorded feedback. See what the policy asked for and what the robot actually did.
-
+
01 / THE FRAMEWORK
Close the loop. Inspect every step.
Connect environment adapters, model policies and local control. Inspect the requested action, the measured motion and the independently recorded result.
+
-
01
Observe
Dual-camera RGB
+
01
Observe
Camera RGB where supported
Robot proprioception
SENSOR INPUT
→
-
02
Decide
Structured GPT actions
-or a local VLA policy
POLICY
→
+
02
Decide
GPT / VLA policies
+Jev text primitives
POLICY
→
03
Act
Validate and track targets
-MuJoCo / LIBERO
LOCAL CONTROL
→
+MuJoCo backendsLOCAL CONTROL
→
04
Check
New images and motion
Execution status and error
FEEDBACK
@@ -73,10 +73,28 @@
See. ∥Independent evaluationObject ground truth, rewards and success signals go to the viewer and result records only.EVALUATOR ≠ POLICY INPUT
- A / INTERFACES
One framework, multiple policies
Use GPT vision and structured output, or run SmolVLA, Diffusion Policy and ACT in an isolated inference environment. Each adapter preserves its observation and action protocol.
Step once, pause after an action, or continue. Inspect the requested TCP target, actual displacement and controller error before allowing the next decision.
Connect GPT vision and structured actions, LIBERO-specific local VLA presets, or Jev text primitives. Each policy keeps its declared inputs and action protocol.
Jog XYZ, rotate Rx/Ry/Rz, or combine translation and rotation in fixed or tool axes. Inspect position and orientation error. Manual control needs no model key.
Backend integration is not a model success result. The existing LeRobot presets remain LIBERO-specific; robosuite and RoboCasa do not yet provide complete control-step video export.
+
+
+
THE LOCAL APPLICATION
A workspace for the whole loop.
Manual control, model runs and scene settings have dedicated panels beside the cameras. These actual local-application screenshots use the built-in Panda tabletop scene; the website does not run their controls.
+
+
01 / MANUAL CONTROL
Fixed or tool axes, translation, rotation and gripper control. Position and orientation feedback stay visible.
+
02 / MODEL RUNS
Connect a policy, inspect its settings, then step or run. Opening a panel does not submit a model request.
+
+
Apply a scene when you are ready
Choose a backend and task, then apply to rebuild the environment. Editing a task instruction does not change the official evaluator. The stop control remains available across panels.
Open 127.0.0.1:8765 for the default tabletop demo. Starting the simulator and using manual controls requires no model key. Use another terminal, or stop the demo with Ctrl+C, before installing LIBERO.
Open 127.0.0.1:8765 for the built-in tabletop workspace. Manual controls need no model key. Use another terminal, or stop with Ctrl+C, before installing an optional backend.
Connect an API service with image and structured-output support, or use the ChatGPT login channel of a recent official Codex CLI. Run diagnostics, then try a small movement. Model requests use your provider or account quota.
Choose GPT through a compatible image and structured-output API or official Codex login, a local LIBERO VLA preset, or Jev text primitives with a separate TypeSafe API key. Diagnose the connection before a small action.
Jev receives explicit instructions and public proprioception, not images. It selects a bounded primitive; local code handles units, coordinates and motion. It does not visually locate objects or provide autonomous grasping.
Model requests use the selected provider or account quota. Switching workspace panels does not trigger inference.
