diff --git a/README.md b/README.md index 343c5ed7..e251d81b 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ This repository is the home of the models: everything for exploring, running, an See it work — zero installVideo generation, visual reasoning1 min Generate my first videoQuickstart ↓10 min Reason over images & videoReasoner notebook10 min - Serve an OpenAI-compatible APIServing setup guide — vLLM, vLLM-Omni, or NIM30 min + Serve models through an APIServing setup guide — vLLM, vLLM-Omni, or the Cosmos3 Certified NIM30 min Post-train on my own data — SFT, distillation, RLCosmos Framework, then evaluate herehours Explore runnable notebooksCookbooksbrowse Evaluate a modelEvaluation suites — PAIBench, Physics-IQ, VLMEvalKithours @@ -164,7 +164,7 @@ Training, optimization, and deployment tooling lives in [Cosmos Framework](https | [Cosmos Curator](https://github.com/NVIDIA/cosmos-curator) | Distributed data curation: processing, annotation, filtering, dedup | | [Cosmos Evaluator](https://github.com/NVIDIA/cosmos-evaluator) | Automated evaluation system for world generation & reasoning outputs | -Cosmos 3 runs on Diffusers, Transformers, vLLM, vLLM-Omni, SGLang, TensorRT-LLM, and NIM — pick a backend in the [environment setup guide](cookbooks/cosmos3/README.md). +Cosmos 3 runs on Diffusers, Transformers, vLLM, vLLM-Omni, SGLang, TensorRT-LLM, and [NIM](cookbooks/cosmos3/nim/README.md) — pick a backend in the [environment setup guide](cookbooks/cosmos3/README.md). ## Community & contributing diff --git a/cookbooks/cosmos3/README.md b/cookbooks/cosmos3/README.md index 9dd21aa5..7f2563d8 100644 --- a/cookbooks/cosmos3/README.md +++ b/cookbooks/cosmos3/README.md @@ -14,8 +14,6 @@ backend you want to run and follow that one section. | [vLLM](#vllm) | OpenAI-compatible reasoning server (image/video understanding) | Reasoner | | [vLLM-Omni](#vllm-omni) | OpenAI-compatible generation server (image/video/audio/action/transfer) | Generator (Audiovisual, Action, **Transfer**) | | [Cosmos3 Certified NIM](#cosmos3-certified-nim) | Prebuilt NGC container serving either the Generator (image/video/action/transfer) or the Reasoner from one image | Reasoner, Generator (Audiovisual, Action, Transfer) | -| [Reasoner NIM](#reasoner-nim) | Prebuilt OpenAI-compatible reasoning server (image/video understanding); no venv | Reasoner | -| [Generator NIM](#generator-nim) | Prebuilt NGC container serving the Cosmos3 Generator for Text-to-Video and Image-to-Video inference | Generator (Audiovisual) | ## Prerequisites @@ -35,9 +33,9 @@ backend you want to run and follow that one section. To disable the guardrail, set `enable_safety_checker=False` (Diffusers), `TRTLLM_DISABLE_COSMOS3_GUARDRAILS=1` or `use_guardrails: false` through `extra_params` (TensorRT-LLM), `guardrails: false` (vLLM-Omni - `extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework). For Generator NIM set environment variables `NIM_ENABLE_TEXT_GUARDRAILS=0 NIM_ENABLE_VIDEO_GUARDRAILS=0`. -- NIMs don't need Hugging Face access; instead, an NGC API key is required - (used as `NGC_API_KEY`). You can generate one on [build.nvidia.com](https://build.nvidia.com/) or [NGC](https://catalog.ngc.nvidia.com/), then run `docker login nvcr.io` once (username `$oauthtoken`, password = your key). This repository uses the Reasoner NIM image `nvcr.io/nim/nvidia/cosmos3-reasoner` and the Generator NIM image `nvcr.io/nim/nvidia/cosmos3-generator`. + `extra_params`/`extra_args`), or `--no-guardrails` (Cosmos Framework). For the NIM Generator runtime, see [Guardrails](nim/operations.md#guardrails). +- NIM doesn't need Hugging Face access; instead, an NGC API key is required + (used as `NGC_API_KEY`). You can generate one on [build.nvidia.com](https://build.nvidia.com/) or [NGC](https://catalog.ngc.nvidia.com/), then run `docker login nvcr.io` once (username `$oauthtoken`, password = your key). Use [NIM](nim/README.md), image `nvcr.io/nim/nvidia/cosmos3:2.0.0`, and select the Generator or Reasoner runtime at launch. - For the Cosmos Framework backend: access to `git@github.com:NVIDIA/cosmos-framework.git`. - Enough local disk for the venv/image, the uv cache, and the model cache. Nano downloads plus CUDA dependencies can take tens of GiB. @@ -779,7 +777,7 @@ Ensure the server has enough GPUs for the product of enabled degrees ## NIM -Prebuilt NGC containers for Cosmos3. Like vLLM-Omni, NIM runs from Docker, so +A prebuilt NGC container image for Cosmos3. Like vLLM-Omni, NIM runs from Docker, so there is no venv or `--torch-backend` to manage. Unlike the Hugging Face based backends, NIM authenticates with an NGC API key instead of a Hugging Face token (see [Prerequisites](#prerequisites)). @@ -791,7 +789,7 @@ export NGC_API_KEY= echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin ``` -Each NIM below exposes readiness at `GET /v1/health/ready` after model +NIM exposes readiness at `GET /v1/health/ready` after model download, engine initialization, and warmup complete. ### Cosmos3 Certified NIM @@ -833,95 +831,13 @@ selects the checkpoint: `nano` or `super` for either runtime, plus `super-i2v-4step` for the Generator. On DGX Spark/GB10 and Jetson AGX Thor, the Reasoner also needs `NIM_GPU_MEMORY_UTILIZATION` set explicitly. -The [Cosmos3 Certified NIM cookbook](nim/README.md) is the full guide — +The [NIM cookbook](nim/README.md) is the full guide — [deployment](nim/deployment.md), [support matrix](nim/support-matrix.md), [configuration](nim/configuration.md), and [API reference](nim/api-reference.md), with worked examples for [generation](nim/generation.md), [action](nim/action.md), [transfer](nim/transfer.md), and [reasoning](nim/reasoning.md). -### Reasoner NIM - -A prebuilt container that serves the Reasoner over an OpenAI-compatible API for -image and video understanding. - -Start a Nano Reasoner server (publishes the API on port 8000; the first run -downloads the model into `~/.cache/nim`): - -```bash -export NGC_API_KEY= - -docker run --runtime=nvidia --gpus all \ - --shm-size=32GB \ - -e NGC_API_KEY="$NGC_API_KEY" \ - -e NIM_MODEL_SIZE=nano \ - -v ~/.cache/nim:/opt/nim/.cache \ - -u $(id -u) \ - -p 8000:8000 \ - nvcr.io/nim/nvidia/cosmos3-reasoner:1.7.0 -``` - -For **Cosmos3-Super-Reasoner** (the larger model), set `-e NIM_MODEL_SIZE=super`. -The container serves `nvidia/cosmos3-nano-reasoner` (or -`nvidia/cosmos3-super-reasoner`); pass that exact name as the request `model`, or -resolve it dynamically with `client.models.list()`. - -### Generator NIM - -A prebuilt container that serves **Cosmos3-Generator Text-to-Video and Image-to-Video -only** through `POST /v1/infer`. The NIM infers mode from the request fields: -non-empty `prompt` with no `image` means Text-to-Video; `image` provided means Image-to-Video. The -response is JSON with a base64-encoded MP4 in `b64_video`. - -It does **not** expose text-to-image, video-to-video, sound/audio generation, -action modes, or transfer controls. Use vLLM-Omni or Cosmos Framework for those -broader Generator workflows. - -Start a Nano Generator server (default `NIM_MODEL_SIZE=nano`, `NIM_PRECISION=fp8`, -`NIM_PERF_PROFILE=latency`): - -```bash -export NGC_API_KEY= -export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}" -mkdir -p "$LOCAL_NIM_CACHE" -chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true - -docker run --runtime=nvidia --gpus all \ - --shm-size=32GB \ - --ulimit nofile=65536:65536 \ - -e NGC_API_KEY="$NGC_API_KEY" \ - -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \ - -p 8000:8000 \ - nvcr.io/nim/nvidia/cosmos3-generator:1.0.0 -``` - -For **Cosmos3-Super Generator**, add `-e NIM_MODEL_SIZE=super`. Other selection -knobs: - -| Env var | Values | Default | Use | -| --- | --- | --- | --- | -| `NIM_MODEL_SIZE` | `nano`, `super` | `nano` | Selects 8B Nano or 32B Super | -| `NIM_PRECISION` | `bf16`, `fp8`, `nvfp4` | `fp8` | Selects precision; `nvfp4` requires Blackwell | -| `NIM_PERF_PROFILE` | `latency`, `throughput` | `latency` | Optimizes profile selection objective | -| `NIM_TAGS_SELECTOR` | comma-separated `key=value` filters | unset | Advanced profile pinning, e.g. `model_size=super,nim_tp=2` | - -A quick T2V smoke test: - -```bash -curl -sS -X POST http://127.0.0.1:8000/v1/infer \ - -H 'Accept: application/json' \ - -H 'Content-Type: application/json' \ - -d '{ - "prompt": "A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components.", - "seed": 42, - "guidance_scale": 6.0, - "steps": 35, - "resolution": "256", - "num_output_frames": 25, - "fps": 24.0 - }' | jq -r '.b64_video' | base64 -d > /tmp/cosmos3_generator_nim_t2v.mp4 -``` - ## Verify the environment For the Cosmos Framework / Diffusers / vLLM venvs, check that PyTorch sees the GPU: diff --git a/cookbooks/cosmos3/generator/audiovisual/README.md b/cookbooks/cosmos3/generator/audiovisual/README.md index 627275a1..744cbac1 100644 --- a/cookbooks/cosmos3/generator/audiovisual/README.md +++ b/cookbooks/cosmos3/generator/audiovisual/README.md @@ -5,26 +5,27 @@ Generate images and video (with optional audio) from text, image or video prompt distilled Cosmos3-Super students across Cosmos Framework, Diffusers, vLLM-Omni, TRT-LLM and NIM backends. Sample prompts live under [`assets/`](./assets). -> **NIM scope:** `Cosmos3-Generator` NIM currently exposes Text2Video and -> Image2Video only. It does not expose text-to-image, video-to-video, -> sound/audio generation, action modes, or transfer controls. +> **NIM scope:** [Cosmos3 Certified NIM](../../nim/README.md) supports +> text-to-image, text-to-video, image-to-video, and video-to-video, +> plus [Action](../../nim/action.md) and [Transfer](../../nim/transfer.md). +> Sound/audio generation is not exposed by the public Generator API. Environment setup for every backend is centralized in the shared [Cosmos3 cookbooks environment setup](../../README.md) guide; each backend below links to the section you need. The quickstarts are minimal text-to-video examples to get one generation running per backend — run them from this folder. -Generator requires the Guardrail. Request access to the gated +For backends other than NIM, request access to the gated [nvidia/Cosmos-1.0-Guardrail](https://huggingface.co/nvidia/Cosmos-1.0-Guardrail) HF repository before running these examples. To disable the guardrail, set `enable_safety_checker=False` (Diffusers), `TRTLLM_DISABLE_COSMOS3_GUARDRAILS=1` or `use_guardrails: false` through `extra_params` (TensorRT-LLM), `guardrails: false` (vLLM-Omni `extra_params`/`extra_args`), or -`--no-guardrails` (Cosmos Framework). For -Generator NIM set `NIM_ENABLE_TEXT_GUARDRAILS=0 NIM_ENABLE_VIDEO_GUARDRAILS=0`. +`--no-guardrails` (Cosmos Framework). For NIM, see +[Guardrails](../../nim/operations.md#guardrails). -NIM backends use NGC authentication instead of Hugging Face login; see the -[Generator NIM setup](../../README.md#generator-nim) for details. +NIM uses NGC authentication instead of Hugging Face login; see the +[NIM setup](../../nim/deployment.md) for details. ## Run with Cosmos Framework @@ -423,125 +424,12 @@ live in the ## Run with NIM -### Quickstart - -Set up the environment: [Generator NIM setup](../../README.md#generator-nim). -`Cosmos3-Generator` NIM is a prebuilt NGC container that serves Text2Video and -Image2Video through `POST /v1/infer`. It returns JSON with a base64-encoded MP4 -in `b64_video`; unlike vLLM-Omni, it does not use `/v1/videos/sync` and does not -return MP4 bytes directly. - -Authenticate Docker to NGC once: - -```bash -export NGC_API_KEY= -echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin -``` - -Launch **Cosmos3-Nano** (default model size, FP8, latency profile): - -```bash -export LOCAL_NIM_CACHE="${LOCAL_NIM_CACHE:-$HOME/.cache/nim}" -mkdir -p "$LOCAL_NIM_CACHE" -chmod -R 777 "$LOCAL_NIM_CACHE" 2>/dev/null || true - -docker run -it --rm --name cosmos3-generator \ - --runtime=nvidia \ - --gpus all \ - --shm-size=32GB \ - --ulimit nofile=65536:65536 \ - -e NGC_API_KEY="$NGC_API_KEY" \ - -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \ - -p 8000:8000 \ - nvcr.io/nim/nvidia/cosmos3-generator:1.0.0 -``` - -Launch **Cosmos3-Super** by adding `NIM_MODEL_SIZE=super`: - -```bash -docker run -it --rm --name cosmos3-generator \ - --runtime=nvidia \ - --gpus all \ - --shm-size=32GB \ - --ulimit nofile=65536:65536 \ - -e NGC_API_KEY="$NGC_API_KEY" \ - -e NIM_MODEL_SIZE=super \ - -e NIM_PRECISION=fp8 \ - -e NIM_PERF_PROFILE=latency \ - -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \ - -p 8000:8000 \ - nvcr.io/nim/nvidia/cosmos3-generator:1.0.0 -``` - -Wait until the readiness endpoint returns success: - -```bash -curl -fsS http://127.0.0.1:8000/v1/health/ready -``` - -Send a Text2Video request. The NIM infers T2V from a non-empty `prompt` with no -`image` field: - -```bash -curl -sS -X POST http://127.0.0.1:8000/v1/infer \ - -H 'Accept: application/json' \ - -H 'Content-Type: application/json' \ - -d '{ - "prompt": "A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components. Photorealistic, cinematic lighting.", - "seed": 42, - "guidance_scale": 6.0, - "steps": 35, - "resolution": "256", - "num_output_frames": 25, - "fps": 24.0 - }' | jq -r '.b64_video' | base64 -d > /tmp/cosmos3_nim_t2v.mp4 -``` - -Send an Image2Video request with a local image encoded as a data URI: - -```bash -IMG_B64=$(base64 -w 0 assets/images/image2video/humanoid_robot.jpg) -cat > /tmp/cosmos3_nim_i2v.json < /tmp/cosmos3_nim_i2v.mp4 -``` - -Key request fields and constraints: - -| Field | Constraint / default | -| --- | --- | -| `prompt` | Required for T2V; optional when `image` is provided; max 20000 chars | -| `negative_prompt` | Optional; omitted means the server uses the Cosmos3 default | -| `image` | I2V conditioning image; raw base64, `data:image/...;base64,...`, or public URL when URL inputs are enabled | -| `guidance_scale` | `1.0` to `7.0`, default `6.0` | -| `steps` | `1` to `100`, default `35` | -| `resolution` | `256`, `480`, `720`, optionally with `_16_9`, `_1_1`, `_9_16`, `_4_3`, or `_3_4` | -| `num_output_frames` | Must follow the 4k+1 cadence (`25, 29, 33, ...`); caps: `256 <= 397`, `480 <= 297`, `720 <= 197` | -| `fps` | `1.0` to `60.0`, recommended `10` to `30`, default `24.0` | - -### Notebook walkthrough - -[`run_with_nim.ipynb`](./run_with_nim.ipynb) launches the NIM container, -waits for readiness, inspects the service endpoints, sends T2V and I2V requests, -decodes `b64_video`, and previews the generated MP4 files inline. - -### Limitations +Use [NIM](../../nim/README.md) with `NIM_MODEL_TYPE=generator`. Follow the +[deployment guide](../../nim/deployment.md) +to select a compatible model and latency or throughput profile, then use the +[generation guide](../../nim/generation.md) for text-to-image, text-to-video, +image-to-video, and video-to-video requests through `POST /v1/infer`. -`Cosmos3-Generator` NIM currently exposes **Text2Video** and **Image2Video** only. -It does **not** expose text-to-image, video-to-video, sound/audio generation, -action modes, or transfer controls. For those broader Generator API workflows, -use vLLM-Omni or Cosmos Framework as appropriate. +Requests select the task explicitly with `model_mode`. The guide includes +[Python examples](../../nim/examples/) for general-purpose and four-step +models, response decoding, and saving generated media. diff --git a/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb b/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb deleted file mode 100644 index bebed76a..00000000 --- a/cookbooks/cosmos3/generator/audiovisual/run_with_nim.ipynb +++ /dev/null @@ -1,397 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "license-header", - "metadata": {}, - "source": [ - "" - ] - }, - { - "cell_type": "markdown", - "id": "intro", - "metadata": {}, - "source": [ - "### Cosmos3 Generator inference with NVIDIA NIM\n", - "\n", - "This notebook runs the prebuilt `Cosmos3-Generator` NIM container and sends video-generation requests to its HTTP API. It mirrors the Reasoner NIM cookbook structure, but uses the Generator NIM API contract:\n", - "\n", - "1. Launch `nvcr.io/nim/nvidia/cosmos3-generator:1.0.0`.\n", - "2. Wait for `GET /v1/health/ready`.\n", - "3. Inspect `/v1/models`, `/v1/metadata`, and `/v1/version`.\n", - "4. Send Text2Video and Image2Video requests to `POST /v1/infer`.\n", - "5. Decode the JSON response field `b64_video` into `.mp4` files and preview them.\n", - "\n", - "The NIM container downloads model artifacts from NGC into a local cache on first launch. You need an `NGC_API_KEY` and Docker login to `nvcr.io` before starting the container.\n", - "\n", - "Useful links:\n", - "- Container [page](https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/cosmos3-generator/-) on NGC\n", - "- [Documentation](https://docs.nvidia.com/nim/cosmos/latest/introduction.html) for the Cosmos NIMs, each page have a table specific to Cosmos3-Generator NIM" - ] - }, - { - "cell_type": "markdown", - "id": "capabilities", - "metadata": {}, - "source": [ - "## Capability scope\n", - "\n", - "`Cosmos3-Generator` NIM currently exposes only the two video-generation modes below. It infers the mode automatically from request fields; there is no explicit `mode` field.\n", - "\n", - "| Workflow | Request shape | Output | Supported here? |\n", - "| --- | --- | --- | --- |\n", - "| Text2Video | non-empty `prompt`, no `image` | JSON with `b64_video` | Yes |\n", - "| Image2Video | `image` plus optional `prompt` | JSON with `b64_video` | Yes |\n", - "| Text-to-image | — | image | No |\n", - "| Video-to-video | — | video | No |\n", - "| Sound/audio generation | — | video + audio | No |\n", - "| Action modes | — | video/action | No |\n", - "| Transfer controls | — | controlled video | No |\n", - "\n", - "The request schema used in this notebook is Cosmos3-specific: `prompt`, `negative_prompt`, `image`, `seed`, `guidance_scale`, `steps`, `resolution`, `num_output_frames`, and `fps`. Unknown request fields are rejected by the server." - ] - }, - { - "cell_type": "markdown", - "id": "launch-intro", - "metadata": {}, - "source": [ - "## 1. Launch the NIM container\n", - "\n", - "Set `NGC_API_KEY` before launching Jupyter, or set it in the notebook environment. Authenticate Docker once with:\n", - "\n", - "```bash\n", - "echo \"$NGC_API_KEY\" | docker login nvcr.io --username '$oauthtoken' --password-stdin\n", - "```\n", - "\n", - "The default launch below uses Nano (`NIM_MODEL_SIZE=nano`), FP8 (`NIM_PRECISION=fp8`), and the latency profile (`NIM_PERF_PROFILE=latency`). Set `NIM_MODEL_SIZE=super` before running the launch cell to serve the larger model." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "helpers", - "metadata": {}, - "outputs": [], - "source": [ - "from pathlib import Path\n", - "import base64\n", - "import json\n", - "import mimetypes\n", - "import os\n", - "import subprocess\n", - "import time\n", - "\n", - "import requests\n", - "from IPython.display import Video, display\n", - "\n", - "\n", - "def find_repo_root() -> Path:\n", - " try:\n", - " return Path(\n", - " subprocess.check_output([\"git\", \"rev-parse\", \"--show-toplevel\"], text=True).strip()\n", - " ).resolve()\n", - " except Exception:\n", - " return Path.cwd().resolve()\n", - "\n", - "\n", - "COSMOS_ROOT = find_repo_root()\n", - "AUDIOVISUAL_ROOT = COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"generator\" / \"audiovisual\"\n", - "ASSETS_ROOT = AUDIOVISUAL_ROOT / \"assets\"\n", - "OUTPUT_DIR = AUDIOVISUAL_ROOT / \"outputs\" / \"cosmos3_generator_nim\"\n", - "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n", - "\n", - "I2V_IMAGE = ASSETS_ROOT / \"images\" / \"image2video\" / \"humanoid_robot.jpg\"\n", - "assert I2V_IMAGE.exists(), I2V_IMAGE\n", - "\n", - "NIM_PORT = int(os.environ.get(\"NIM_PORT\", \"8000\"))\n", - "BASE_URL = f\"http://127.0.0.1:{NIM_PORT}\"\n", - "\n", - "\n", - "def file_data_uri(path: Path) -> str:\n", - " mime = mimetypes.guess_type(path.name)[0] or \"application/octet-stream\"\n", - " encoded = base64.b64encode(path.read_bytes()).decode(\"ascii\")\n", - " return f\"data:{mime};base64,{encoded}\"\n", - "\n", - "\n", - "def write_b64_video(response_json: dict, path: Path) -> Path:\n", - " data = response_json[\"b64_video\"]\n", - " path.write_bytes(base64.b64decode(data))\n", - " return path\n", - "\n", - "\n", - "def post_infer(payload: dict, output_name: str) -> Path:\n", - " response = requests.post(\n", - " f\"{BASE_URL}/v1/infer\",\n", - " json=payload,\n", - " headers={\"Accept\": \"application/json\", \"Content-Type\": \"application/json\"},\n", - " timeout=60 * 60,\n", - " )\n", - " response.raise_for_status()\n", - " output_path = OUTPUT_DIR / output_name\n", - " return write_b64_video(response.json(), output_path)\n", - "\n", - "\n", - "print(\"cosmos root:\", COSMOS_ROOT)\n", - "print(\"audiovisual root:\", AUDIOVISUAL_ROOT)\n", - "print(\"i2v image:\", I2V_IMAGE)\n", - "print(\"outputs:\", OUTPUT_DIR)\n", - "print(\"NIM base URL:\", BASE_URL)\n", - "print(\"I2V data URI prefix:\", file_data_uri(I2V_IMAGE)[:64] + \"...\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "launch-docker", - "metadata": {}, - "outputs": [], - "source": [ - "%%bash\n", - "set -euo pipefail\n", - "\n", - ": \"${NGC_API_KEY:?Set NGC_API_KEY before launching the Cosmos3-Generator NIM}\"\n", - "\n", - "export CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n", - "export IMG_NAME=\"${IMG_NAME:-nvcr.io/nim/nvidia/cosmos3-generator:1.0.0}\"\n", - "export NIM_MODEL_SIZE=\"${NIM_MODEL_SIZE:-nano}\"\n", - "export NIM_PRECISION=\"${NIM_PRECISION:-fp8}\"\n", - "export NIM_PERF_PROFILE=\"${NIM_PERF_PROFILE:-latency}\"\n", - "export LOCAL_NIM_CACHE=\"${LOCAL_NIM_CACHE:-$HOME/.cache/nim}\"\n", - "export NIM_PORT=\"${NIM_PORT:-8000}\"\n", - "mkdir -p \"$LOCAL_NIM_CACHE\"\n", - "chmod -R 777 \"$LOCAL_NIM_CACHE\" 2>/dev/null || true\n", - "\n", - "# The container name must be free. If a previous run is still up, stop it first:\n", - "# docker stop nvidia-cosmos3-generator\n", - "docker run -d --rm --name=\"$CONTAINER_NAME\" \\\n", - " --runtime=nvidia \\\n", - " --gpus all \\\n", - " --shm-size=32GB \\\n", - " --ulimit nofile=65536:65536 \\\n", - " -e NGC_API_KEY=\"$NGC_API_KEY\" \\\n", - " -e NIM_MODEL_SIZE=\"$NIM_MODEL_SIZE\" \\\n", - " -e NIM_PRECISION=\"$NIM_PRECISION\" \\\n", - " -e NIM_PERF_PROFILE=\"$NIM_PERF_PROFILE\" \\\n", - " -v \"$LOCAL_NIM_CACHE:/opt/nim/.cache\" \\\n", - " -p \"${NIM_PORT}:8000\" \\\n", - " \"$IMG_NAME\"\n", - "\n", - "echo \"NIM container '$CONTAINER_NAME' launching on port $NIM_PORT\"\n", - "echo \"model_size=$NIM_MODEL_SIZE precision=$NIM_PRECISION profile=$NIM_PERF_PROFILE\"\n" - ] - }, - { - "cell_type": "markdown", - "id": "wait-intro", - "metadata": {}, - "source": [ - "## 2. Wait for readiness\n", - "\n", - "The first run downloads model artifacts and may spend additional time compiling/warming the engine. The server is ready for inference when `GET /v1/health/ready` returns HTTP 200." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "wait-ready", - "metadata": {}, - "outputs": [], - "source": [ - "%%bash\n", - "set -euo pipefail\n", - "\n", - "PORT=\"${NIM_PORT:-8000}\"\n", - "CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n", - "\n", - "echo \"Waiting for Cosmos3-Generator NIM on port ${PORT}...\"\n", - "\n", - "for i in $(seq 1 3600); do\n", - " if curl -fsS \"http://127.0.0.1:${PORT}/v1/health/ready\" >/dev/null 2>&1; then\n", - " echo \"NIM server is ready.\"\n", - " exit 0\n", - " fi\n", - " if ! docker ps --format '{{.Names}}' | grep -q \"^${CONTAINER_NAME}$\"; then\n", - " echo \"Container '${CONTAINER_NAME}' is no longer running. Recent logs:\"\n", - " docker logs --tail 160 \"$CONTAINER_NAME\" 2>&1 || true\n", - " exit 1\n", - " fi\n", - " if (( i % 30 == 0 )); then\n", - " echo \"Still waiting... recent logs:\"\n", - " docker logs --tail 20 \"$CONTAINER_NAME\" 2>&1 || true\n", - " fi\n", - " sleep 2\n", - "done\n", - "\n", - "echo \"Timed out waiting for NIM server. Recent logs:\"\n", - "docker logs --tail 160 \"$CONTAINER_NAME\" 2>&1 || true\n", - "exit 1\n" - ] - }, - { - "cell_type": "markdown", - "id": "inspect-intro", - "metadata": {}, - "source": [ - "## 3. Inspect the running service\n", - "\n", - "`Cosmos3-Generator` exposes standard NIM health/metadata endpoints plus `/v1/version` and a live OpenAPI schema at `/openapi.json`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "inspect-service", - "metadata": {}, - "outputs": [], - "source": [ - "for endpoint in [\"/v1/models\", \"/v1/metadata\", \"/v1/version\"]:\n", - " response = requests.get(f\"{BASE_URL}{endpoint}\", timeout=30)\n", - " response.raise_for_status()\n", - " print(f\"\\n### {endpoint}\")\n", - " print(json.dumps(response.json(), indent=2)[:4000])\n" - ] - }, - { - "cell_type": "markdown", - "id": "schema-intro", - "metadata": {}, - "source": [ - "## 4. Request parameters\n", - "\n", - "Useful constraints for `POST /v1/infer`:\n", - "\n", - "| Field | Constraint / default |\n", - "| --- | --- |\n", - "| `guidance_scale` | float in `[1.0, 7.0]`, default `6.0` |\n", - "| `steps` | int in `[1, 100]`, default `35` |\n", - "| `resolution` | `256`, `480`, `720`, optionally suffixed with `_16_9`, `_1_1`, `_9_16`, `_4_3`, `_3_4`; bare tiers mean 16:9 |\n", - "| `num_output_frames` | must follow the `4k+1` cadence: `25, 29, 33, ...`; per-tier caps are `256 <= 397`, `480 <= 297`, `720 <= 197` |\n", - "| `fps` | float in `[1.0, 60.0]`, recommended `10` to `30`, default `24.0` |\n", - "\n", - "The examples use `resolution=\"256\"` and `num_output_frames=25` so a smoke test completes faster. Increase these values for quality once the workflow is verified." - ] - }, - { - "cell_type": "markdown", - "id": "t2v-intro", - "metadata": {}, - "source": [ - "## 5. Text2Video\n", - "\n", - "For Text2Video, provide a non-empty `prompt` and omit `image`. The response is JSON; decode `response[\"b64_video\"]` to get the MP4 bytes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "t2v-request", - "metadata": {}, - "outputs": [], - "source": [ - "t2v_payload = {\n", - " \"prompt\": \"A humanoid robot walks through a futuristic warehouse, inspecting shelves of mechanical components. Photorealistic, cinematic lighting.\",\n", - " \"seed\": 42,\n", - " \"guidance_scale\": 6.0,\n", - " \"steps\": 35,\n", - " \"resolution\": \"256\",\n", - " \"num_output_frames\": 25,\n", - " \"fps\": 24.0,\n", - "}\n", - "\n", - "t2v_path = post_infer(t2v_payload, \"cosmos3_generator_nim_t2v.mp4\")\n", - "print(\"wrote\", t2v_path, t2v_path.stat().st_size, \"bytes\")\n", - "display(Video(str(t2v_path), embed=True, html_attributes=\"controls loop\"))\n" - ] - }, - { - "cell_type": "markdown", - "id": "i2v-intro", - "metadata": {}, - "source": [ - "## 6. Image2Video\n", - "\n", - "For Image2Video, provide an `image` field. The field accepts raw base64, a `data:image/...;base64,...` URI, or a public URL when URL inputs are enabled. This example uses a checked-in local image encoded as a data URI, so the NIM container does not need to see the host filesystem." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "i2v-request", - "metadata": {}, - "outputs": [], - "source": [ - "i2v_payload = {\n", - " \"prompt\": \"The humanoid robot performs a controlled standing backflip in a modern living room, then lands steadily on both feet.\",\n", - " \"image\": file_data_uri(I2V_IMAGE),\n", - " \"seed\": 123,\n", - " \"guidance_scale\": 6.0,\n", - " \"steps\": 35,\n", - " \"resolution\": \"256\",\n", - " \"num_output_frames\": 25,\n", - " \"fps\": 24.0,\n", - "}\n", - "\n", - "i2v_path = post_infer(i2v_payload, \"cosmos3_generator_nim_i2v.mp4\")\n", - "print(\"wrote\", i2v_path, i2v_path.stat().st_size, \"bytes\")\n", - "display(Video(str(i2v_path), embed=True, html_attributes=\"controls loop\"))\n" - ] - }, - { - "cell_type": "markdown", - "id": "playback-note", - "metadata": {}, - "source": [ - "## Playback note\n", - "\n", - "The generated `b64_video` payload is an MP4 container. Some environments may not preview VP9-in-MP4 inline. If the notebook preview does not play, verify the written file with a player such as `ffplay`, `mpv`, or IINA, or re-encode externally to H.264." - ] - }, - { - "cell_type": "markdown", - "id": "cleanup-intro", - "metadata": {}, - "source": [ - "## 7. Optional cleanup\n", - "\n", - "Stop the detached container when you are done. The cached artifacts stay in `LOCAL_NIM_CACHE` for future launches." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "cleanup", - "metadata": {}, - "outputs": [], - "source": [ - "%%bash\n", - "set -euo pipefail\n", - "CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-generator}\"\n", - "docker stop \"$CONTAINER_NAME\"\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/cookbooks/cosmos3/reasoner/README.md b/cookbooks/cosmos3/reasoner/README.md index 89a1cd1e..3aba238f 100644 --- a/cookbooks/cosmos3/reasoner/README.md +++ b/cookbooks/cosmos3/reasoner/README.md @@ -140,66 +140,14 @@ example resolves the model dynamically ## Run with NIM -### Quickstart - -Set up the environment: [NIM setup](../README.md#nim). That launches the -prebuilt [Cosmos 3 Reasoner NIM](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/cosmos3-reasoner) -container, which serves the same OpenAI-compatible chat-completions API as the -vLLM path on port 8000 — without the vLLM/CUDA setup. You can also try this same -NIM interactively in your browser on the -[cosmos3-nano-reasoner build page](https://build.nvidia.com/nvidia/cosmos3-nano-reasoner). - -Once the server is up, query it with the OpenAI client (the served model name is -`nvidia/cosmos3-nano-reasoner`, or `nvidia/cosmos3-super-reasoner` for Super): - -```python -import base64 -import mimetypes -from pathlib import Path -import openai - -image_path = Path("assets/robot_153.jpg").resolve() -mime = mimetypes.guess_type(image_path.name)[0] or "application/octet-stream" -image_url = f"data:{mime};base64,{base64.b64encode(image_path.read_bytes()).decode('ascii')}" - -client = openai.OpenAI(api_key="not-used", base_url="http://127.0.0.1:8000/v1") - -response = client.chat.completions.create( - model="nvidia/cosmos3-nano-reasoner", - messages=[ - { - "role": "user", - "content": [ - {"type": "image_url", "image_url": {"url": image_url}}, - {"type": "text", "text": "Caption the image in detail."}, - ], - } - ], - max_tokens=4096, -) - -print(response.choices[0].message.content) -``` - -Video inputs use the `video_url` content type and accept `media_io_kwargs` -frame-sampling controls through `extra_body` -(`{"media_io_kwargs": {"video": {"fps": 4.0}}}`). See the -[Cosmos Reason 3 NIM API reference](https://docs.nvidia.com/nim/vision-language-models/1.7.0/examples/cosmos-reason3/api.html) -for the full request reference. - -### Notebook walkthrough - -[`run_with_nim.ipynb`](./run_with_nim.ipynb) is the NIM counterpart to the vLLM -notebook: it launches the NIM container, waits for readiness, and then runs the -same image and video examples — detailed captioning, VQA, temporal localization, -embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, -action CoT trajectories, driving scenes, physical-plausibility, and situation -understanding. Because the container does not see the host filesystem, local -assets are sent as base64 data URIs; video frame sampling is controlled with -`media_io_kwargs` while spatial resolution is controlled with -`mm_processor_kwargs`. Each example resolves the served model dynamically -(`client.models.list()`), so the prompts work unchanged for both the `nano` and -`super` sizes. +Use [Cosmos3 Certified NIM](../nim/README.md) with +`NIM_MODEL_TYPE=reasoner`. Follow the [deployment guide](../nim/deployment.md) +to select a compatible model and hardware configuration, then use the +[reasoning guide](../nim/reasoning.md) for image and video requests through +the OpenAI-compatible Chat Completions API. + +The guide includes [Python examples](../nim/examples/reasoner.py) for +captioning, visual question answering, grounding, and other reasoning tasks. ## Run with Transformers diff --git a/cookbooks/cosmos3/reasoner/run_with_nim.ipynb b/cookbooks/cosmos3/reasoner/run_with_nim.ipynb deleted file mode 100644 index ed9b32b6..00000000 --- a/cookbooks/cosmos3/reasoner/run_with_nim.ipynb +++ /dev/null @@ -1,1225 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "license-header", - "metadata": {}, - "source": [ - "" - ] - }, - { - "cell_type": "markdown", - "id": "b1f96e7d", - "metadata": {}, - "source": [ - "### Cosmos3 Reasoner inference with NVIDIA NIM\n", - "\n", - "This notebook:\n", - "1. Launches the [Cosmos 3 Reasoner NIM](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/cosmos3-reasoner) container, a prebuilt, optimized, OpenAI-compatible server.\n", - "2. Waits for it to become ready.\n", - "3. Sends the same image and video reasoning requests with the `openai` client.\n", - "\n", - "Compared to the vLLM notebook, there is no Python environment or CUDA-pairing to set up — the NIM container ships everything. The container serves two sizes, selected with `NIM_MODEL_SIZE`:\n", - "\n", - "| `NIM_MODEL_SIZE` | Served model name |\n", - "| --- | --- |\n", - "| `nano` (default) | `nvidia/cosmos3-nano-reasoner` |\n", - "| `super` | `nvidia/cosmos3-super-reasoner` |\n", - "\n", - "You can also try this same NIM interactively in your browser on the [cosmos3-nano-reasoner build page](https://build.nvidia.com/nvidia/cosmos3-nano-reasoner). See the [Cosmos Reason 3 NIM API reference](https://docs.nvidia.com/nim/vision-language-models/1.7.0/examples/cosmos-reason3/api.html) for the full request reference." - ] - }, - { - "cell_type": "markdown", - "id": "99497707", - "metadata": {}, - "source": [ - "## 1. Launch the NIM container\n", - "\n", - "You need an `NGC_API_KEY` (from [build.nvidia.com](https://build.nvidia.com/nvidia/cosmos3-nano-reasoner) or [NGC](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/cosmos3-reasoner)) set in the environment **before** starting Jupyter, or set it from Python with `os.environ[\"NGC_API_KEY\"] = \"nvapi-...\"`. Docker must also be logged in to `nvcr.io` (`docker login nvcr.io`, username `$oauthtoken`, password = your key).\n", - "\n", - "The first launch downloads and caches the model into `~/.cache/nim`, which can take a while; later launches start quickly." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "b286f1b7", - "metadata": {}, - "outputs": [], - "source": [ - "from pathlib import Path\n", - "import os\n", - "import subprocess\n", - "\n", - "\n", - "def find_repo_root() -> Path:\n", - " try:\n", - " return Path(\n", - " subprocess.check_output([\"git\", \"rev-parse\", \"--show-toplevel\"], text=True).strip()\n", - " ).resolve()\n", - " except Exception:\n", - " return Path.cwd().resolve()\n", - "\n", - "\n", - "COSMOS_ROOT = find_repo_root()\n", - "COSMOS_REASONER_ASSETS = COSMOS_ROOT / \"cookbooks\" / \"cosmos3\" / \"reasoner\" / \"assets\"\n", - "\n", - "assert COSMOS_REASONER_ASSETS.exists(), COSMOS_REASONER_ASSETS\n", - "\n", - "\n", - "def asset_path(name: str) -> Path:\n", - " path = COSMOS_REASONER_ASSETS / name\n", - " if not path.exists():\n", - " raise FileNotFoundError(path)\n", - " return path\n", - "\n", - "\n", - "import base64\n", - "import mimetypes\n", - "\n", - "\n", - "def asset_data_uri(name: str) -> str:\n", - " \"\"\"Read a local asset and return a base64 ``data:`` URI.\n", - "\n", - " The NIM container does not see the host filesystem, so local images and\n", - " videos are inlined as base64 data URIs rather than passed as ``file://``\n", - " paths. The examples use local assets; public URLs work too.\n", - " \"\"\"\n", - " path = asset_path(name)\n", - " mime = mimetypes.guess_type(path.name)[0] or \"application/octet-stream\"\n", - " encoded = base64.b64encode(path.read_bytes()).decode(\"ascii\")\n", - " return f\"data:{mime};base64,{encoded}\"\n", - "\n", - "\n", - "print(\"cosmos root:\", COSMOS_ROOT)\n", - "print(\"Reasoner assets:\", COSMOS_REASONER_ASSETS)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "d999ab52", - "metadata": {}, - "outputs": [], - "source": [ - "%%bash\n", - "set -euo pipefail\n", - "\n", - ": \"${NGC_API_KEY:?Set NGC_API_KEY (from build.nvidia.com / NGC) before launching}\"\n", - "\n", - "export CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-reasoner}\"\n", - "export IMG_NAME=\"${IMG_NAME:-nvcr.io/nim/nvidia/cosmos3-reasoner:1.7.0}\"\n", - "# Set NIM_MODEL_SIZE=super for the larger Cosmos3-Super-Reasoner.\n", - "export NIM_MODEL_SIZE=\"${NIM_MODEL_SIZE:-nano}\"\n", - "export LOCAL_NIM_CACHE=\"${LOCAL_NIM_CACHE:-$HOME/.cache/nim}\"\n", - "export NIM_PORT=\"${NIM_PORT:-8000}\"\n", - "mkdir -p \"$LOCAL_NIM_CACHE\"\n", - "\n", - "# The container name must be free. If a previous run is still up, stop it first:\n", - "# docker stop nvidia-cosmos3-reasoner\n", - "# Detached (-d) so the notebook keeps control; logs are read by the next cell.\n", - "docker run -d --rm --name=\"$CONTAINER_NAME\" \\\n", - " --runtime=nvidia \\\n", - " --gpus all \\\n", - " --shm-size=32GB \\\n", - " -e NGC_API_KEY=\"$NGC_API_KEY\" \\\n", - " -e NIM_MODEL_SIZE=\"$NIM_MODEL_SIZE\" \\\n", - " -v \"$LOCAL_NIM_CACHE:/opt/nim/.cache\" \\\n", - " -u \"$(id -u)\" \\\n", - " -p \"${NIM_PORT}:8000\" \\\n", - " \"$IMG_NAME\"\n", - "\n", - "echo \"NIM container '$CONTAINER_NAME' (size=$NIM_MODEL_SIZE) launching on port $NIM_PORT\"\n" - ] - }, - { - "cell_type": "markdown", - "id": "265a72d4", - "metadata": {}, - "source": [ - "## 2. Wait for the server\n", - "\n", - "The next cell polls the NIM readiness endpoint and streams recent container logs. The first run downloads the model, so allow several minutes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a5f45c3d", - "metadata": {}, - "outputs": [], - "source": [ - "%%bash\n", - "set -euo pipefail\n", - "\n", - "PORT=\"${NIM_PORT:-8000}\"\n", - "CONTAINER_NAME=\"${CONTAINER_NAME:-nvidia-cosmos3-reasoner}\"\n", - "\n", - "echo \"Waiting for NIM server on port ${PORT} (first run downloads the model)...\"\n", - "\n", - "for i in $(seq 1 3600); do\n", - " if curl -fsS \"http://127.0.0.1:${PORT}/v1/health/ready\" >/dev/null 2>&1; then\n", - " echo \"NIM server is ready.\"\n", - " exit 0\n", - " fi\n", - " if ! docker ps --format '{{.Names}}' | grep -q \"^${CONTAINER_NAME}$\"; then\n", - " echo \"Container '${CONTAINER_NAME}' is no longer running. Recent logs:\"\n", - " docker logs --tail 120 \"$CONTAINER_NAME\" 2>&1 || true\n", - " exit 1\n", - " fi\n", - " sleep 2\n", - "done\n", - "\n", - "echo \"Timed out waiting for NIM server. Recent logs:\"\n", - "docker logs --tail 120 \"$CONTAINER_NAME\" 2>&1 || true\n", - "exit 1\n" - ] - }, - { - "cell_type": "markdown", - "id": "8b6c2d78", - "metadata": {}, - "source": [ - "## 3. Query the model\n", - "\n", - "Local images and videos are sent as base64 data URIs (the container does not see the host filesystem). Each request resolves the served model dynamically with `client.models.list()`, so the examples work unchanged for both the `nano` and `super` sizes." - ] - }, - { - "cell_type": "markdown", - "id": "410f97ba", - "metadata": {}, - "source": [ - "### Image Caption" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3022311f", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from IPython.display import Image, display\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "\n", - "image_path = asset_path(\"robot_153.jpg\")\n", - "image_url = asset_data_uri(image_path.name)\n", - "\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n", - " {\"type\": \"text\", \"text\": \"Caption the image in detail.\"},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " seed=0,\n", - ")\n", - "display(Image(filename=str(image_path), width=512))\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "77326560", - "metadata": {}, - "source": [ - "### Video Caption" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7f1ea8e7", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "\n", - "prompt = \"Describe the video in detail.\"\n", - "\n", - "# Plain filesystem path (used for display)\n", - "video_path = str(asset_path(\"video_caption.mp4\"))\n", - "# base64 data URI (used for the model request)\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "# Display the input video (plain path, not the base64 data URI sent to the model)\n", - "display(Video(video_path, embed=True, width=640))\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "87d32bd6", - "metadata": {}, - "source": [ - "### Temporal Localization" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "010a21ca", - "metadata": {}, - "outputs": [], - "source": [ - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "\n", - "# Plain filesystem path (used for display)\n", - "video_path = str(asset_path(\"temporal_localization_1.mp4\"))\n", - "\n", - "display(Video(video_path, embed=True, width=640))\n", - "\n", - "import openai\n", - "\n", - "prompt = (\n", - " \"\"\"List all action segments in the video. For each detected event, you must determine:\n", - "\n", - "Provide the result in json format with 'seconds' for time depiction for each event. Use keywords 'start', 'end' and 'caption' in the json output. Please list multiple events if applicable.\n", - "\n", - "```json\n", - "[\n", - "{\n", - " \"start\": t_start,\n", - " \"end\": t_end,\n", - " \"caption\": EVENT1\n", - "},\n", - "{\n", - " \"start\": t_start,\n", - " \"end\": t_end,\n", - " \"caption\": EVENT2\n", - "},\n", - "...\n", - "]\n", - "``` \"\"\"\n", - ")\n", - "video_url = asset_data_uri(\"temporal_localization_1.mp4\")\n", - "\n", - "client = openai.OpenAI(\n", - " api_key=\"not-used\",\n", - " base_url=\"http://localhost:8000/v1\",\n", - ")\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "0a76e9f5", - "metadata": {}, - "outputs": [], - "source": [ - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "\n", - "# Plain filesystem path (used for display)\n", - "video_path = str(asset_path(\"temporal_localization_2.mp4\"))\n", - "\n", - "display(Video(video_path, embed=True, width=640))" - ] - }, - { - "cell_type": "markdown", - "id": "809e9c57", - "metadata": {}, - "source": [ - "#### Event Timeline" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "61b206c0", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "\n", - "prompt = (\n", - " \"Describe the notable events in the provided video. Provide the result in json format with 'mm:ss.ff' format for time depiction for each event.\"\n", - " \"Use keywords 'start', 'end' and 'caption' in the json output.\"\n", - ")\n", - "video_url = asset_data_uri(\"temporal_localization_2.mp4\")\n", - "\n", - "client = openai.OpenAI(\n", - " api_key=\"not-used\",\n", - " base_url=\"http://localhost:8000/v1\",\n", - ")\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "bbb6d43e", - "metadata": {}, - "source": [ - "#### Timestamp Query" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "40bfb46e", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "\n", - "prompt = \"\"\"When is \"A man in a white sweater walks out of a room carrying a box, closes the door behind him, walks on the floor, and turns left at the end near the wall.\" depicted in the video? Please provide the result in json format with 'mm:ss.ff' format for time depiction for the event. Use keywords 'start', 'end' in the json output.\"\"\"\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "f1fbc5ed", - "metadata": {}, - "source": [ - "#### Interval Question" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "b8d26975", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "\n", - "prompt = \"What happened between 00:05.64 and 00:17.49?\"\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "ca919a80", - "metadata": {}, - "source": [ - "### Embodied Reasoning\n", - "#### Robotics Next Action" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "05a7bbe6", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "\n", - "prompt = \"What can be the next immediate action? Answer the question using the following format: Your reasoning. Write your final answer immediately after the tag.\"\n", - "\n", - "# Plain filesystem path (used for display)\n", - "video_path = str(asset_path(\"robotics_next_action.mp4\"))\n", - "# base64 data URI (used for the model request)\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "# Display the input video (plain path, not the base64 data URI sent to the model)\n", - "display(Video(video_path, embed=True, width=640))\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "b79e6056", - "metadata": {}, - "source": [ - "#### Drive Scene Next Action" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c8393d28", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "\n", - "prompt = \"You are an autonomous vehicle planning system. The video depicts the observation from the vehicle's camera. You need to observe the critical objects in the environment and reason your next action and the driving trajectory ahead.\"\n", - "\n", - "# Plain filesystem path (used for display)\n", - "video_path = str(asset_path(\"drive_scene_next_action.mp4\"))\n", - "# base64 data URI (used for the model request)\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "# Display the input video (plain path, not the base64 data URI sent to the model)\n", - "display(Video(video_path, embed=True, width=640))\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "1f97ac7c", - "metadata": {}, - "source": [ - "#### Robot Planning" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7a2377d3", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "import re\n", - "import openai\n", - "from pathlib import Path\n", - "from PIL import Image as PILImage, ImageDraw\n", - "from IPython.display import display\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "\n", - "image_path = str(asset_path(\"robot_planning.png\"))\n", - "image_url = asset_data_uri(Path(image_path).name) # base64 data URI for the model\n", - "\n", - "# Display the input image (scaled down to fit the cell)\n", - "preview = PILImage.open(image_path).convert(\"RGB\")\n", - "preview.thumbnail((768, 768))\n", - "display(preview)\n", - "\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n", - " {\"type\": \"text\", \"text\": 'The task is to put flower into the red bottle. Generate a plan consisting of subtasks for accomplish the task.'},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " seed=0,\n", - ")\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "4c163252", - "metadata": {}, - "source": [ - "#### Assisted Task Next Action" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3c6147d2", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "\n", - "prompt = \"\"\"This is the overall task that the agent is trying to complete: \"The student exchanges the black ink cartridge of the printer.\"\n", - " In the video, the agent is trying to follow the instruction (a single step out of many to complete the overall task): \"place old ink_cartridge.\"\n", - " What should be the next action of the agent?\n", - " Answer the question using the following format:\n", - " \n", - " Your reasoning.\n", - " \n", - " Write your final answer immediately after the tag.\"\"\"\n", - "\n", - "# Plain filesystem path (used for display)\n", - "video_path = str(asset_path(\"assisted_task_next_action.mp4\"))\n", - "# base64 data URI (used for the model request)\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "# Display the input video (plain path, not the base64 data URI sent to the model)\n", - "display(Video(video_path, embed=True, width=640))\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "1b956864", - "metadata": {}, - "source": [ - "### Common Sense Reasoning" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "50b25fa6", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "\n", - "prompt = \"\"\"Can the countertop support the weight of the juicers?\n", - " Answer the question using the following format:\n", - "\n", - " \n", - " Your reasoning.\n", - " \n", - "\n", - " Write your final answer immediately after the tag.\"\"\"\n", - "\n", - "# Plain filesystem path (used for display)\n", - "video_path = str(asset_path(\"common_sense_reasoning.mp4\"))\n", - "# base64 data URI (used for the model request)\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "\n", - "response = client.chat.completions.create(\n", - " model=client.models.list().data[0].id,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " },\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\"media_io_kwargs\": {\"video\": {\"fps\": 4.0}}},\n", - ")\n", - "\n", - "# Display the input video (plain path, not the base64 data URI sent to the model)\n", - "display(Video(video_path, embed=True, width=640))\n", - "\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "60835463", - "metadata": {}, - "source": [ - "### 2D Grounding" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "53a97880", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "import re\n", - "import openai\n", - "from pathlib import Path\n", - "from PIL import Image as PILImage, ImageDraw\n", - "from IPython.display import display\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "\n", - "image_path = str(asset_path(\"grounding_2d.png\"))\n", - "image_url = asset_data_uri(Path(image_path).name) # base64 data URI for the model\n", - "\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n", - " {\"type\": \"text\", \"text\": \"Locate the accurate bounding box of the load as a whole. Return a json.\"},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " seed=0,\n", - ")\n", - "out = response.choices[0].message.content\n", - "print(out)\n", - "\n", - "\n", - "def parse_boxes(text):\n", - " \"\"\"Pull a JSON array/object of boxes out of the model text (handles ``` fences).\"\"\"\n", - " text = text.strip()\n", - " text = re.sub(r\"^```(?:json)?|```$\", \"\", text, flags=re.MULTILINE).strip()\n", - " m = re.search(r\"\\[.*\\]|\\{.*\\}\", text, re.DOTALL)\n", - " data = json.loads(m.group(0) if m else text)\n", - " return data if isinstance(data, list) else [data]\n", - "\n", - "\n", - "# Draw boxes; coords are normalized to 0-1000\n", - "img = PILImage.open(image_path).convert(\"RGB\")\n", - "W, H = img.size\n", - "draw = ImageDraw.Draw(img)\n", - "\n", - "for obj in parse_boxes(out):\n", - " box = obj.get(\"bbox_2d\") or obj.get(\"bbox\") or obj.get(\"box\")\n", - " if not box:\n", - " continue\n", - " x1, y1, x2, y2 = box\n", - " x1, x2 = x1 / 1000 * W, x2 / 1000 * W\n", - " y1, y2 = y1 / 1000 * H, y2 / 1000 * H\n", - " draw.rectangle([x1, y1, x2, y2], outline=\"red\", width=3)\n", - " label = obj.get(\"label\") or obj.get(\"name\")\n", - " if label:\n", - " draw.text((x1, max(0, y1 - 12)), str(label), fill=\"red\")\n", - "\n", - "# Display scaled down so a large image fits the cell\n", - "preview = img.copy()\n", - "preview.thumbnail((768, 768))\n", - "display(preview)" - ] - }, - { - "cell_type": "markdown", - "id": "16ed27e9", - "metadata": {}, - "source": [ - "### Describe Anything" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8a73b225", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "import re\n", - "import openai\n", - "from pathlib import Path\n", - "from PIL import Image as PILImage, ImageDraw\n", - "from IPython.display import display\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "\n", - "image_path = str(asset_path(\"describe_anything.png\"))\n", - "image_url = asset_data_uri(Path(image_path).name) # base64 data URI for the model\n", - "\n", - "# Display the input image (scaled down to fit the cell)\n", - "preview = PILImage.open(image_path).convert(\"RGB\")\n", - "preview.thumbnail((768, 768))\n", - "display(preview)\n", - "\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n", - " {\"type\": \"text\", \"text\": 'Please caption the notable attributes in the provided image. List and describe all marked subjects in the image with their categories and detailed captions using a json with keyword \"subject_id\", \"category\" and \"caption\".'},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " seed=0,\n", - ")\n", - "print(response.choices[0].message.content)\n" - ] - }, - { - "cell_type": "markdown", - "id": "303ae5af", - "metadata": {}, - "source": [ - "### Action CoT" - ] - }, - { - "cell_type": "markdown", - "id": "bd9e82ea", - "metadata": {}, - "source": [ - "#### Trajectory Coordinates" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7a020759", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "import re\n", - "import openai\n", - "from pathlib import Path\n", - "from PIL import Image as PILImage, ImageDraw\n", - "from IPython.display import display\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "\n", - "image_path = str(asset_path(\"action_cot_trajectory.png\"))\n", - "image_url = asset_data_uri(Path(image_path).name)\n", - "\n", - "prompt = \"\"\"You are given the task \"Move the pink bowl to the right\". Specify the 2D trajectory your end effector should follow in pixel space. Return the trajectory coordinates in JSON format like this: {\"point_2d\": [x, y], \"label\": \"gripper trajectory\"}.\n", - "Answer the question using the following format:\n", - "\n", - "\n", - "Your reasoning.\n", - "\n", - "\n", - "Write your final answer immediately after the tag.\n", - "\"\"\"\n", - "\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " temperature=0.6,\n", - " top_p=0.95,\n", - " presence_penalty=0.0,\n", - " extra_body={\"top_k\": 20, \"repetition_penalty\": 1.0},\n", - ")\n", - "out = response.choices[0].message.content\n", - "print(out)\n", - "\n", - "\n", - "def parse_points(text):\n", - " \"\"\"Grab the JSON list of {point_2d, label} after the tag.\"\"\"\n", - " if \"\" in text:\n", - " text = text.split(\"\")[-1]\n", - " text = re.sub(r\"```(?:json)?\", \"\", text).strip().strip(\"`\").strip()\n", - " m = re.search(r\"\\[.*\\]\", text, re.DOTALL)\n", - " data = json.loads(m.group(0) if m else text)\n", - " return data if isinstance(data, list) else [data]\n", - "\n", - "\n", - "# Visualize the trajectory (points are in pixel space)\n", - "img = PILImage.open(image_path).convert(\"RGB\")\n", - "draw = ImageDraw.Draw(img)\n", - "W, H = img.size\n", - "\n", - "# coords are normalized to 0-1000 (per-axis) -> scale to pixels\n", - "pts = [(o[\"point_2d\"][0] / 1000 * W, o[\"point_2d\"][1] / 1000 * H)\n", - " for o in parse_points(out) if isinstance(o, dict) and \"point_2d\" in o]\n", - "if len(pts) > 1:\n", - " draw.line(pts, fill=\"lime\", width=5)\n", - "for i, (x, y) in enumerate(pts):\n", - " r = 12\n", - " draw.ellipse([x - r, y - r, x + r, y + r], fill=\"red\", outline=\"white\", width=3)\n", - " draw.text((x + 14, y - 14), str(i), fill=\"yellow\")\n", - "preview = img.copy()\n", - "preview.thumbnail((900, 900))\n", - "display(preview)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3351c7e4", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "import re\n", - "import openai\n", - "from pathlib import Path\n", - "from PIL import Image as PILImage, ImageDraw\n", - "from IPython.display import display\n", - "\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "\n", - "image_path = str(asset_path(\"robot_planning.png\"))\n", - "image_url = asset_data_uri(Path(image_path).name)\n", - "\n", - "prompt = \"\"\"You are given the task \"Put flower into the red bottle\". Specify the 2D trajectory your end effector should follow in pixel space. Return the trajectory coordinates in JSON format like this: {\"point_2d\": [x, y], \"label\": \"gripper trajectory\"}. \n", - "Answer the question using the following format:\n", - "\n", - " Your reasoning. \n", - "Write your final answer immediately after the tag.\n", - "\"\"\"\n", - "\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " temperature=0.6,\n", - " top_p=0.95,\n", - " presence_penalty=0.0,\n", - " extra_body={\"top_k\": 20, \"repetition_penalty\": 1.0},\n", - ")\n", - "out = response.choices[0].message.content\n", - "print(out)\n", - "\n", - "\n", - "def parse_points(text):\n", - " \"\"\"Grab the JSON list of {point_2d, label} after the tag.\"\"\"\n", - " if \"\" in text:\n", - " text = text.split(\"\")[-1]\n", - " text = re.sub(r\"```(?:json)?\", \"\", text).strip().strip(\"`\").strip()\n", - " m = re.search(r\"\\[.*\\]\", text, re.DOTALL)\n", - " data = json.loads(m.group(0) if m else text)\n", - " return data if isinstance(data, list) else [data]\n", - "\n", - "\n", - "# Visualize the trajectory (points are in pixel space)\n", - "img = PILImage.open(image_path).convert(\"RGB\")\n", - "draw = ImageDraw.Draw(img)\n", - "W, H = img.size\n", - "\n", - "# coords are normalized to 0-1000 (per-axis) -> scale to pixels\n", - "pts = [(o[\"point_2d\"][0] / 1000 * W, o[\"point_2d\"][1] / 1000 * H)\n", - " for o in parse_points(out) if isinstance(o, dict) and \"point_2d\" in o]\n", - "if len(pts) > 1:\n", - " draw.line(pts, fill=\"lime\", width=5)\n", - "for i, (x, y) in enumerate(pts):\n", - " r = 12\n", - " draw.ellipse([x - r, y - r, x + r, y + r], fill=\"red\", outline=\"white\", width=3)\n", - " draw.text((x + 14, y - 14), str(i), fill=\"yellow\")\n", - "preview = img.copy()\n", - "preview.thumbnail((900, 900))\n", - "display(preview)" - ] - }, - { - "cell_type": "markdown", - "id": "6cb3d76e", - "metadata": {}, - "source": [ - "#### Driving Scene" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2d92a055", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "video_path = str(asset_path(\"action_cot_driving_scene.mp4\"))\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "prompt = \"\"\"The video depicts the observation from the vehicle's camera. You need to think step by step and identify the objects in the scene that are critical for safe navigation.\n", - "Answer the question using the following format:\n", - "\n", - "Your reasoning.\n", - "\n", - "Write your final answer immediately after the tag.\"\"\"\n", - "# Show the input video\n", - "display(Video(video_path, embed=True, width=640))\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " temperature=0.6,\n", - " top_p=0.95,\n", - " presence_penalty=0.0,\n", - " extra_body={\n", - " \"top_k\": 20,\n", - " \"repetition_penalty\": 1.0,\n", - " \"media_io_kwargs\": {\"video\": {\"fps\": 4.0}},\n", - " },\n", - ")\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "f9c09ca4", - "metadata": {}, - "source": [ - "### Physical Plausibility Analysis" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fd80d461", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "video_path = str(asset_path(\"physical_plausibility.mp4\"))\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "prompt = \"\"\"Is this video physically plausible/possible according to your understanding of e.g. object permanence, shape constancy (objects maintain shape over time), continuous trajectories of objects? Assume it is the normal laws of physics.\n", - "Your answer should be based on the events in the video and ignore the quality of the simulation engine. The rising wall is part of the experiment setup and should not be judged for plausibility.\n", - "(A) Possible\n", - "(B) Impossible\"\"\"\n", - "# Show the input video\n", - "display(Video(video_path, embed=True, width=640))\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\n", - " \"media_io_kwargs\": {\"video\": {\"fps\": 4.0}},\n", - " },\n", - ")\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "markdown", - "id": "4fa05fab", - "metadata": {}, - "source": [ - "### Situation Understanding" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c663f0c4", - "metadata": {}, - "outputs": [], - "source": [ - "import openai\n", - "from pathlib import Path\n", - "from IPython.display import Video, display\n", - "client = openai.OpenAI(api_key=\"not-used\", base_url=\"http://localhost:8000/v1\")\n", - "MODEL = client.models.list().data[0].id\n", - "video_path = str(asset_path(\"situation_understanding.mp4\"))\n", - "video_url = asset_data_uri(Path(video_path).name)\n", - "prompt = \"What is the person doing with the skillet? What will the person likely do next in this situation?\"\n", - "# Show the input video\n", - "display(Video(video_path, embed=True, width=640))\n", - "response = client.chat.completions.create(\n", - " model=MODEL,\n", - " messages=[\n", - " {\n", - " \"role\": \"user\",\n", - " \"content\": [\n", - " {\"type\": \"video_url\", \"video_url\": {\"url\": video_url}},\n", - " {\"type\": \"text\", \"text\": prompt},\n", - " ],\n", - " }\n", - " ],\n", - " max_tokens=4096,\n", - " extra_body={\n", - " \"media_io_kwargs\": {\"video\": {\"fps\": 4.0}},\n", - " },\n", - ")\n", - "print(response.choices[0].message.content)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "27b44dde", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/cookbooks/cosmos3/reasoner/run_with_transformers.ipynb b/cookbooks/cosmos3/reasoner/run_with_transformers.ipynb index 327cee17..e8282df7 100644 --- a/cookbooks/cosmos3/reasoner/run_with_transformers.ipynb +++ b/cookbooks/cosmos3/reasoner/run_with_transformers.ipynb @@ -497,7 +497,7 @@ " by changing the prompt and asset — see the\n", " [Reasoner Prompt Guide](./reasoner_prompt_guide.md).\n", "- Need an OpenAI-compatible server instead of in-process Python? See\n", - " [`run_with_vllm.ipynb`](./run_with_vllm.ipynb) or [`run_with_nim.ipynb`](./run_with_nim.ipynb).\n" + " [`run_with_vllm.ipynb`](./run_with_vllm.ipynb) or the [NIM reasoning guide](../nim/reasoning.md).\n" ] } ], diff --git a/inference_benchmarks.md b/inference_benchmarks.md index 79487ee5..2c730706 100644 --- a/inference_benchmarks.md +++ b/inference_benchmarks.md @@ -1,12 +1,12 @@ # Inference Benchmarks -These reports collect inference benchmarks for Cosmos3. **Generator** pages measure image, video, audiovisual, and action-generation latency across PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM where results are available. **Reasoner** sections below measure VLM serving and token-generation performance for text, image, and video inputs through vLLM and Hugging Face Transformers. +These reports collect inference benchmarks for Cosmos3. **Generator** pages measure image, video, audiovisual, and action-generation latency across PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and [Cosmos3 Certified NIM](cookbooks/cosmos3/nim/README.md) (NIM) where results are available. **Reasoner** sections below measure VLM serving and token-generation performance for text, image, and video inputs through vLLM and Hugging Face Transformers. -Results are published incrementally from internal benchmark runs. **Empty cells mean that combination has not been measured yet** — not that it is unsupported. See each model page or section for workload details and data-source definitions. +Results are published incrementally from internal benchmark runs. Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. See each model page or section for workload details and data-source definitions. ## Generator benchmarks -Generator results are organized by model so future checkpoints can be added without expanding this page into another wall of tables. Use the section links for modality-oriented navigation. The FP8 pages cover the quantized Nano and Super checkpoints; every other Generator page reports BF16, except NIM rows, which are FP8. +Generator results are organized by model so future checkpoints can be added without expanding this page into another wall of tables. Use the section links for modality-oriented navigation. The FP8 pages cover the quantized Nano and Super checkpoints; every other Generator page reports BF16, except NIM rows, which use FP8 with output quality validated to be comparable to the BF16 baseline. | Model | Primary vision generation | Additional audiovisual generation | Transfer generation | Action generation | |---|---|---|---|---| diff --git a/inference_benchmarks/cosmos3-nano-fp8-generator.md b/inference_benchmarks/cosmos3-nano-fp8-generator.md index 9690dc1f..d27a4f94 100644 --- a/inference_benchmarks/cosmos3-nano-fp8-generator.md +++ b/inference_benchmarks/cosmos3-nano-fp8-generator.md @@ -33,7 +33,9 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -PyTorch values are from PBR `#309161`, **FP8 Cosmos3-Nano/Super (regular non-distilled) Generator PyTorch Inference Benchmarking**, reporting average generation latency over datacenter configurations at 1, 4, and 8 GPUs. Diffusers values are from PBR `#309434`, **FP8 Cosmos3-Nano/Super Generator Diffusers Inference Benchmarking**, measured on a single GPU only and at 480p and 720p only, which is why the 256p column is empty for Diffusers and why its rows are empty wherever a block reports eight GPUs. Neither campaign covered H100 NVL or H200 NVL, so those GPUs are absent from this page. GB300 appears only in the Diffusers campaign. vLLM-Omni, TensorRT-LLM, and NIM have no FP8 results yet. Values are rounded to two decimal places. +PyTorch values are from PBR `#309161`, **FP8 Cosmos3-Nano/Super (regular non-distilled) Generator PyTorch Inference Benchmarking**, reporting average generation latency over datacenter configurations at 1, 4, and 8 GPUs. Diffusers values are from PBR `#309434`, **FP8 Cosmos3-Nano/Super Generator Diffusers Inference Benchmarking**, measured on a single GPU only and at 480p and 720p only, which is why the 256p column is empty for Diffusers and why its rows are empty wherever a block reports eight GPUs. Neither OSS campaign covered H100 NVL or H200 NVL; those GPUs have NIM results only in the primary tables. GB300 appears only in the Diffusers campaign. vLLM-Omni and TensorRT-LLM have no FP8 results here. Values are rounded to two decimal places. + +[NIM](../cookbooks/cosmos3/nim/README.md) results cover t2v, i2v, and t2i. NIM reports `Avg. Generation Time (s)` using FP8 latency profiles with offload disabled, excluding request overhead and MP4 encoding. Runs use concurrency 1, three measured requests, and 189 video frames or one image frame. The same measurements are also shown on the [Nano page](cosmos3-nano-generator.md), which includes [additional GPU counts and profiles](cosmos3-nano-generator.md#additional-nim-configurations). Precision alone does not establish identical checkpoints or benchmark settings across backends. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -63,18 +65,27 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 4.71 | 47.34 | 173.02 | | | Diffusers | 1 | | | | +| | NIM | 1 | 7.24 | 81.35 | 313.63 | | **H20** | PyTorch | 1 | 11.82 | 112.34 | 427.62 | | | Diffusers | 1 | | | | +| | NIM | 1 | 18.78 | 195.35 | 776.37 | | **H100 80GB HBM3** | PyTorch | 1 | 3.12 | 26.01 | 94.01 | | | Diffusers | 1 | | 69.55 | 237.29 | +| | NIM | 1 | 5.35 | 46.08 | 173.20 | | **H200 141GB HBM3** | PyTorch | 1 | 3.09 | 26.59 | 94.57 | | | Diffusers | 1 | | 68.32 | 237.18 | +| | NIM | 1 | 5.02 | 46.93 | 177.23 | | **B200** | PyTorch | 1 | 1.97 | 14.85 | 51.85 | | | Diffusers | 1 | | 39.70 | 124.10 | +| | NIM | 1 | 2.94 | 25.03 | 91.62 | | **B300** | PyTorch | 1 | 1.99 | 13.68 | 46.41 | | | Diffusers | 1 | | 37.61 | 115.14 | +| | NIM | 1 | 2.86 | 23.67 | 85.52 | | **GB300** | PyTorch | 1 | | | | | | Diffusers | 1 | | 34.53 | 97.30 | +| | NIM | 1 | | | | +| **H100 NVL** | NIM | 1 | 6.70 | 67.09 | 252.22 | +| **H200 NVL** | NIM | 1 | 5.42 | 56.85 | 227.34 | ### Image-to-Video (i2v) @@ -82,18 +93,27 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 4.50 | 45.96 | 169.15 | | | Diffusers | 1 | | | | +| | NIM | 1 | 7.27 | 81.50 | 313.64 | | **H20** | PyTorch | 1 | 11.19 | 110.19 | 419.99 | | | Diffusers | 1 | | | | +| | NIM | 1 | 18.69 | 195.87 | 776.48 | | **H100 80GB HBM3** | PyTorch | 1 | 2.96 | 25.04 | 91.65 | | | Diffusers | 1 | | 71.06 | 241.53 | +| | NIM | 1 | 5.37 | 46.34 | 173.14 | | **H200 141GB HBM3** | PyTorch | 1 | 2.94 | 25.56 | 92.44 | | | Diffusers | 1 | | 69.69 | 237.95 | +| | NIM | 1 | 5.07 | 46.95 | 177.57 | | **B200** | PyTorch | 1 | 1.87 | 14.24 | 50.24 | | | Diffusers | 1 | | 40.69 | 126.07 | +| | NIM | 1 | 2.98 | 25.59 | 91.65 | | **B300** | PyTorch | 1 | 1.78 | 13.08 | 44.94 | | | Diffusers | 1 | | 38.35 | 117.18 | +| | NIM | 1 | 2.88 | 23.74 | 85.50 | | **GB300** | PyTorch | 1 | | | | | | Diffusers | 1 | | 34.96 | 99.37 | +| | NIM | 1 | | | | +| **H100 NVL** | NIM | 1 | 6.76 | 67.50 | 251.21 | +| **H200 NVL** | NIM | 1 | 5.43 | 57.02 | 226.12 | ### Text-to-Image (t2i) @@ -101,21 +121,30 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 0.97 | 1.02 | 1.50 | | | Diffusers | 1 | | 7.29 | | +| | NIM | 1 | 1.01 | 1.33 | 2.25 | | **H20** | PyTorch | 1 | 1.27 | 1.74 | 3.25 | | | Diffusers | 1 | | | | +| | NIM | 1 | 1.18 | 2.54 | 4.70 | | **H100 80GB HBM3** | PyTorch | 1 | 1.16 | 1.14 | 1.21 | | | Diffusers | 1 | | 7.50 | 7.68 | +| | NIM | 1 | 1.10 | 1.24 | 1.51 | | **H200 141GB HBM3** | PyTorch | 1 | 1.12 | 1.14 | 1.21 | | | Diffusers | 1 | | 7.44 | 7.74 | +| | NIM | 1 | 1.17 | 1.18 | 1.42 | | **B200** | PyTorch | 1 | 0.96 | 0.92 | 0.97 | | | Diffusers | 1 | | 6.82 | 6.94 | +| | NIM | 1 | 0.99 | 1.06 | 1.02 | | **B300** | PyTorch | 1 | 0.95 | 0.95 | 1.51 | | | Diffusers | 1 | | 10.28 | 6.71 | +| | NIM | 1 | 1.45 | 1.50 | 1.48 | | **GB300** | PyTorch | 1 | | | | | | Diffusers | 1 | | 11.79 | 13.19 | +| | NIM | 1 | | | | +| **H100 NVL** | NIM | 1 | 1.14 | 1.15 | 1.72 | +| **H200 NVL** | NIM | 1 | 1.12 | 1.16 | 1.46 | Notes: -1. All times measured on identical workloads (same seed, sampler settings, prompt). +1. OSS times measured on identical workloads (same seed, sampler settings, prompt). 2. Multi-GPU configurations use tensor parallelism. 3. Values are average generation latency in seconds; lower is better. 4. The **GPUs** column gives the number of GPUs behind every value in that row, and is the same for every engine in a GPU block so the rows can be compared directly. diff --git a/inference_benchmarks/cosmos3-nano-generator.md b/inference_benchmarks/cosmos3-nano-generator.md index f508f38b..db82ad3f 100644 --- a/inference_benchmarks/cosmos3-nano-generator.md +++ b/inference_benchmarks/cosmos3-nano-generator.md @@ -4,7 +4,7 @@ These tables report **Cosmos3-Nano Generator** latency in seconds. Lower is better. -Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. Cosmos3-Nano fits on a single GPU at 720p, so all populated rows here report one GPU. +Each primary vision-generation row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. Cosmos3-Nano fits on a single GPU at 720p, so its primary vision-generation rows report one GPU. Additional NIM GPU counts are listed separately. Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. @@ -16,6 +16,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso - [Text-to-Video (t2v)](#text-to-video-t2v) - [Image-to-Video (i2v)](#image-to-video-i2v) - [Text-to-Image (t2i)](#text-to-image-t2i) +- [Additional NIM configurations](#additional-nim-configurations) - [Additional audiovisual generation](#additional-audiovisual-generation) - [Video-to-Video (v2v)](#video-to-video-v2v) - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) @@ -34,7 +35,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. +The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch values preserve the previously published campaigns. NIM values report average generation time using FP8 latency profiles (see note 7). NIM FP8 output quality has been validated to be comparable to the BF16 baseline. OSS rows use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. TensorRT-LLM values are from PBR `#308000`, **Cosmos3-Generator TRT-LLM Inference Benchmarking**, which sweeps tensor-parallel, CFG-parallel, and Ulysses-parallel splits at each GPU count; the fastest split at the count a row reports is the one published. That campaign covers six GPUs, so the TensorRT-LLM rows for H100 NVL and H200 NVL are empty, and it includes no action workloads. Values are rounded to two decimal places. @@ -66,42 +67,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | 1 | 10.65 | 105.61 | 371.53 | | | Diffusers | 1 | 11.20 | 112.00 | 392.00 | | | TensorRT-LLM | 1 | 10.27 | 102.55 | 369.78 | -| | NIM | 1 | 7.93 | 82.69 | 318.69 | +| | NIM | 1 | 7.24 | 81.35 | 313.63 | | **H20** | PyTorch | 1 | 30.57 | 257.51 | 931.39 | | | vLLM-Omni | 1 | 28.58 | 256.97 | 929.81 | | | Diffusers | 1 | 30.20 | 258.00 | 926.00 | | | TensorRT-LLM | 1 | 26.92 | 251.79 | 908.90 | -| | NIM | 1 | 17.81 | 192.07 | 771.37 | +| | NIM | 1 | 18.78 | 195.35 | 776.37 | | **H100 NVL** | PyTorch | 1 | 10.03 | 84.12 | 297.27 | | | vLLM-Omni | 1 | 9.25 | 80.75 | 281.99 | | | Diffusers | 1 | 11.00 | 90.00 | 324.20 | | | TensorRT-LLM | — | | | | -| | NIM | 1 | 7.09 | 68.01 | 267.73 | +| | NIM | 1 | 6.70 | 67.09 | 252.22 | | **H200 NVL** | PyTorch | 1 | 8.17 | 69.79 | 244.39 | | | vLLM-Omni | 1 | 7.44 | 64.58 | 240.05 | | | Diffusers | 1 | 9.00 | 74.00 | 276.20 | | | TensorRT-LLM | — | | | | -| | NIM | 1 | 5.87 | 57.10 | 229.63 | +| | NIM | 1 | 5.42 | 56.85 | 227.34 | | **H100 80GB HBM3** | PyTorch | 1 | 7.61 | 59.83 | 207.78 | | | vLLM-Omni | 1 | 6.97 | 58.17 | 202.29 | | | Diffusers | 1 | 9.00 | 68.00 | 240.00 | | | TensorRT-LLM | 1 | 6.50 | 60.10 | 223.51 | -| | NIM | 1 | 5.72 | 51.73 | 199.46 | +| | NIM | 1 | 5.35 | 46.08 | 173.20 | | **H200 141GB HBM3** | PyTorch | 1 | 7.53 | 60.18 | 214.28 | | | vLLM-Omni | 1 | 6.79 | 58.14 | 208.36 | | | Diffusers | 1 | 9.00 | 67.00 | 239.60 | | | TensorRT-LLM | 1 | 6.42 | 60.57 | 225.06 | -| | NIM | 1 | 5.72 | 51.90 | 200.63 | +| | NIM | 1 | 5.02 | 46.93 | 177.23 | | **B200** | PyTorch | 1 | 4.56 | 33.20 | 114.85 | | | vLLM-Omni | 1 | 4.03 | 32.04 | 107.79 | | | Diffusers | 1 | 7.00 | 36.80 | 117.00 | | | TensorRT-LLM | 1 | 3.43 | 28.28 | 100.02 | -| | NIM | 1 | 3.72 | 26.68 | 93.33 | +| | NIM | 1 | 2.94 | 25.03 | 91.62 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 4.46 | 32.18 | 102.10 | | | Diffusers | 1 | 6.40 | 39.32 | 121.74 | | | TensorRT-LLM | 1 | 3.31 | 26.52 | 91.15 | -| | NIM | 1 | 4.51 | 27.49 | 90.39 | +| | NIM | 1 | 2.86 | 23.67 | 85.52 | ### Image-to-Video (i2v) @@ -111,42 +112,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | 1 | 11.04 | 107.77 | 376.44 | | | Diffusers | 1 | 12.00 | 112.00 | 397.00 | | | TensorRT-LLM | 1 | 10.77 | 105.03 | 374.89 | -| | NIM | 1 | 8.23 | 84.58 | 326.96 | +| | NIM | 1 | 7.27 | 81.50 | 313.64 | | **H20** | PyTorch | 1 | 31.36 | 257.10 | 933.07 | | | vLLM-Omni | 1 | 29.50 | 261.56 | 940.16 | | | Diffusers | 1 | 31.00 | 258.00 | 925.00 | | | TensorRT-LLM | 1 | 28.07 | 254.79 | 915.71 | -| | NIM | 1 | 18.67 | 195.10 | 774.88 | +| | NIM | 1 | 18.69 | 195.87 | 776.48 | | **H100 NVL** | PyTorch | 1 | 10.19 | 84.50 | 298.57 | | | vLLM-Omni | 1 | 9.62 | 82.61 | 286.19 | | | Diffusers | 1 | 11.00 | 91.00 | 325.20 | | | TensorRT-LLM | — | | | | -| | NIM | 1 | 7.39 | 69.39 | 272.29 | +| | NIM | 1 | 6.76 | 67.50 | 251.21 | | **H200 NVL** | PyTorch | 1 | 8.27 | 69.99 | 246.62 | | | vLLM-Omni | 1 | 7.83 | 66.39 | 243.52 | | | Diffusers | 1 | 9.00 | 74.00 | 275.20 | | | TensorRT-LLM | — | | | | -| | NIM | 1 | 6.25 | 58.64 | 232.47 | +| | NIM | 1 | 5.43 | 57.02 | 226.12 | | **H100 80GB HBM3** | PyTorch | 1 | 7.64 | 59.95 | 207.87 | | | vLLM-Omni | 1 | 7.37 | 59.77 | 205.97 | | | Diffusers | 1 | 9.00 | 68.00 | 239.80 | | | TensorRT-LLM | 1 | 6.90 | 63.24 | 226.15 | -| | NIM | 1 | 6.08 | 53.02 | 202.59 | +| | NIM | 1 | 5.37 | 46.34 | 173.14 | | **H200 141GB HBM3** | PyTorch | 1 | 7.65 | 60.51 | 214.80 | | | vLLM-Omni | 1 | 7.28 | 59.64 | 209.65 | | | Diffusers | 1 | 9.00 | 67.20 | 240.00 | | | TensorRT-LLM | 1 | 6.74 | 63.54 | 231.15 | -| | NIM | 1 | 6.04 | 53.25 | 203.66 | +| | NIM | 1 | 5.07 | 46.95 | 177.57 | | **B200** | PyTorch | 1 | 4.60 | 33.08 | 113.90 | | | vLLM-Omni | 1 | 4.33 | 33.09 | 110.08 | | | Diffusers | 1 | 7.00 | 37.00 | 116.00 | | | TensorRT-LLM | 1 | 3.70 | 29.33 | 102.35 | -| | NIM | 1 | 4.05 | 27.72 | 95.57 | +| | NIM | 1 | 2.98 | 25.59 | 91.65 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 5.61 | 33.45 | 104.75 | | | Diffusers | 1 | 13.40 | 49.59 | 131.25 | | | TensorRT-LLM | 1 | 3.70 | 27.73 | 93.88 | -| | NIM | 1 | 4.50 | 28.90 | 92.59 | +| | NIM | 1 | 2.88 | 23.74 | 85.50 | ### Text-to-Image (t2i) @@ -156,43 +157,106 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | 1 | 1.59 | 2.87 | 4.99 | | | Diffusers | 1 | 2.00 | 4.00 | 5.00 | | | TensorRT-LLM | 1 | 1.18 | 2.35 | 4.50 | +| | NIM | 1 | 1.01 | 1.33 | 2.25 | | **H20** | PyTorch | 1 | 3.06 | 6.51 | 12.31 | | | vLLM-Omni | 1 | 1.73 | 4.92 | 10.73 | | | Diffusers | 1 | 3.00 | 6.00 | 10.00 | | | TensorRT-LLM | 1 | 1.94 | 5.55 | 11.56 | +| | NIM | 1 | 1.18 | 2.54 | 4.70 | | **H100 NVL** | PyTorch | 1 | 2.77 | 2.83 | 4.21 | | | vLLM-Omni | 1 | 1.55 | 1.92 | 3.43 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | | | TensorRT-LLM | — | | | | +| | NIM | 1 | 1.14 | 1.15 | 1.72 | | **H200 NVL** | PyTorch | 1 | 2.75 | 2.85 | 3.58 | | | vLLM-Omni | 1 | 1.53 | 1.58 | 2.81 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | | | TensorRT-LLM | — | | | | +| | NIM | 1 | 1.12 | 1.16 | 1.46 | | **H100 80GB HBM3** | PyTorch | 1 | 3.01 | 3.01 | 3.45 | | | vLLM-Omni | 1 | 1.61 | 1.53 | 2.61 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | | | TensorRT-LLM | 1 | 1.02 | 1.30 | 2.63 | +| | NIM | 1 | 1.10 | 1.24 | 1.51 | | **H200 141GB HBM3** | PyTorch | 1 | 2.96 | 3.04 | 3.28 | | | vLLM-Omni | 1 | 1.57 | 1.52 | 2.60 | | | Diffusers | 1 | 3.00 | 3.00 | 4.00 | | | TensorRT-LLM | 1 | 1.05 | 1.30 | 2.60 | +| | NIM | 1 | 1.17 | 1.18 | 1.42 | | **B200** | PyTorch | 1 | 2.68 | 2.75 | 2.87 | | | vLLM-Omni | 1 | 1.49 | 1.20 | 1.76 | | | Diffusers | 1 | 3.00 | 3.20 | 3.00 | | | TensorRT-LLM | 1 | 0.86 | 0.88 | 1.40 | +| | NIM | 1 | 0.99 | 1.06 | 1.02 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 1.97 | 1.81 | 2.34 | | | Diffusers | 1 | 3.86 | 3.86 | 3.99 | | | TensorRT-LLM | 1 | 1.36 | 1.30 | 1.40 | +| | NIM | 1 | 1.45 | 1.50 | 1.48 | Notes: -1. All times measured on identical workloads (same seed, sampler settings, prompt). +1. OSS times measured on identical workloads (same seed, sampler settings, prompt). 2. Multi-GPU configurations use tensor parallelism, except TensorRT-LLM, which may also split across CFG and Ulysses dimensions. 3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo and are subject to change before GA; the H100 NVL values in particular are pre-release. 4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs. Single-GPU values come from PBR `#308202` and PBR `#308451`; multi-GPU values from PBR `#308918`. 5. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. 6. At 256p, multi-GPU configurations on B300 may underperform single-GPU because of small-workload tensor-parallel overhead, so single-GPU is the recommended deployment at this resolution. -7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. +7. NIM numbers use FP8 latency profiles with offload disabled and report `Avg. Generation Time (s)`, excluding request overhead and MP4 encoding. Runs use concurrency 1, three measured requests, and 189 video frames or one image frame. + +## Additional NIM configurations + +Average generation time in seconds for GPU counts not shown in the primary tables above. These runs use the same FP8 latency profiles with offload disabled (see note 7). + +| GPU | Modality | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| RTX PRO 6000 Blackwell | t2v | 4 | 3.45 | 25.41 | 90.81 | +| RTX PRO 6000 Blackwell | t2v | 8 | 2.83 | 17.04 | 49.33 | +| H20 | t2v | 4 | 5.88 | 53.77 | 203.35 | +| H20 | t2v | 8 | 3.84 | 28.51 | 105.11 | +| H100 NVL | t2v | 4 | 2.50 | 24.35 | 102.48 | +| H100 NVL | t2v | 8 | 2.46 | 12.96 | 48.50 | +| H200 NVL | t2v | 4 | 2.62 | 16.95 | 62.77 | +| H200 NVL | t2v | 8 | 2.14 | 10.30 | 33.53 | +| H100 80GB HBM3 | t2v | 4 | 2.24 | 13.78 | 48.35 | +| H100 80GB HBM3 | t2v | 8 | 1.81 | 7.77 | 26.14 | +| H200 141GB HBM3 | t2v | 4 | 2.12 | 13.65 | 48.75 | +| H200 141GB HBM3 | t2v | 8 | 1.85 | 7.56 | 25.78 | +| B200 | t2v | 4 | 1.57 | 7.54 | 25.97 | +| B200 | t2v | 8 | 1.56 | 4.41 | 14.50 | +| B300 | t2v | 4 | 2.48 | 7.36 | 24.41 | +| B300 | t2v | 8 | 2.50 | 4.82 | 13.80 | +| RTX PRO 6000 Blackwell | i2v | 4 | 3.50 | 25.54 | 90.98 | +| RTX PRO 6000 Blackwell | i2v | 8 | 2.98 | 16.03 | 49.97 | +| H20 | i2v | 4 | 5.97 | 53.73 | 203.76 | +| H20 | i2v | 8 | 3.96 | 28.66 | 105.06 | +| H100 NVL | i2v | 4 | 2.61 | 24.29 | 99.98 | +| H100 NVL | i2v | 8 | 2.53 | 13.18 | 48.97 | +| H200 NVL | i2v | 4 | 2.72 | 17.07 | 62.81 | +| H200 NVL | i2v | 8 | 2.32 | 10.58 | 33.63 | +| H100 80GB HBM3 | i2v | 4 | 2.29 | 13.88 | 48.33 | +| H100 80GB HBM3 | i2v | 8 | 1.93 | 7.92 | 26.37 | +| H200 141GB HBM3 | i2v | 4 | 2.19 | 13.72 | 48.66 | +| H200 141GB HBM3 | i2v | 8 | 1.95 | 7.69 | 26.00 | +| B200 | i2v | 4 | 1.62 | 7.62 | 26.13 | +| B200 | i2v | 8 | 1.71 | 4.59 | 14.50 | +| B300 | i2v | 4 | 2.48 | 7.45 | 24.56 | +| B300 | i2v | 8 | 2.62 | 4.97 | 14.20 | +| RTX PRO 6000 Blackwell | t2i | 4 | 0.90 | 0.92 | 1.13 | +| RTX PRO 6000 Blackwell | t2i | 8 | 0.89 | 0.94 | 1.00 | +| H20 | t2i | 4 | 1.06 | 1.13 | 1.61 | +| H20 | t2i | 8 | 1.09 | 2.98 | 1.18 | +| H100 NVL | t2i | 4 | 0.96 | 0.98 | 1.00 | +| H100 NVL | t2i | 8 | 0.97 | 2.72 | 1.00 | +| H200 NVL | t2i | 4 | 0.92 | 0.98 | 0.98 | +| H200 NVL | t2i | 8 | 1.03 | 2.81 | 1.04 | +| H100 80GB HBM3 | t2i | 4 | 0.98 | 1.00 | 1.08 | +| H100 80GB HBM3 | t2i | 8 | 1.02 | 2.93 | 1.07 | +| H200 141GB HBM3 | t2i | 4 | 1.04 | 0.99 | 1.06 | +| H200 141GB HBM3 | t2i | 8 | 1.09 | 2.93 | 1.10 | +| B200 | t2i | 4 | 0.87 | 0.91 | 0.89 | +| B200 | t2i | 8 | 0.90 | 1.02 | 0.91 | +| B300 | t2i | 4 | 1.36 | 1.40 | 1.37 | +| B300 | t2i | 8 | 1.38 | 1.45 | 1.47 | ## Additional audiovisual generation diff --git a/inference_benchmarks/cosmos3-super-fp8-generator.md b/inference_benchmarks/cosmos3-super-fp8-generator.md index 75b34a6d..983651e0 100644 --- a/inference_benchmarks/cosmos3-super-fp8-generator.md +++ b/inference_benchmarks/cosmos3-super-fp8-generator.md @@ -33,7 +33,9 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -PyTorch values are from PBR `#309161`, **FP8 Cosmos3-Nano/Super (regular non-distilled) Generator PyTorch Inference Benchmarking**, reporting average generation latency over datacenter configurations at 1, 4, and 8 GPUs. Diffusers values are from PBR `#309434`, **FP8 Cosmos3-Nano/Super Generator Diffusers Inference Benchmarking**, measured on a single GPU only and at 480p and 720p only, which is why the 256p column is empty for Diffusers and why its rows are empty wherever a block reports eight GPUs. Neither campaign covered H100 NVL or H200 NVL, so those GPUs are absent from this page. GB300 appears only in the Diffusers campaign. vLLM-Omni, TensorRT-LLM, and NIM have no FP8 results yet. Values are rounded to two decimal places. +PyTorch values are from PBR `#309161`, **FP8 Cosmos3-Nano/Super (regular non-distilled) Generator PyTorch Inference Benchmarking**, reporting average generation latency over datacenter configurations at 1, 4, and 8 GPUs. Diffusers values are from PBR `#309434`, **FP8 Cosmos3-Nano/Super Generator Diffusers Inference Benchmarking**, measured on a single GPU only and at 480p and 720p only, which is why the 256p column is empty for Diffusers and why its rows are empty wherever a block reports eight GPUs. Neither OSS campaign covered H100 NVL or H200 NVL; those GPUs have NIM results only in the primary tables. GB300 appears only in the Diffusers campaign. vLLM-Omni and TensorRT-LLM have no FP8 results here. Values are rounded to two decimal places. + +[NIM](../cookbooks/cosmos3/nim/README.md) results cover t2v, i2v, and t2i. NIM reports `Avg. Generation Time (s)` using FP8 latency profiles with offload disabled, excluding request overhead and MP4 encoding. Runs use concurrency 1, three measured requests, and 189 video frames or one image frame. The same measurements are also shown on the [Super page](cosmos3-super-generator.md), which includes [additional GPU counts and profiles](cosmos3-super-generator.md#additional-nim-configurations). Precision alone does not establish identical checkpoints or benchmark settings across backends. These reports establish the reported timing matrix but do not expose every prompt and action payload in this repository. The linked public recipes explain modality behavior and provide representative payloads; their example-specific frame counts and action chunk sizes should not be treated as the exact internal benchmark inputs. @@ -63,18 +65,27 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 15.27 | 154.79 | 589.57 | | | Diffusers | 1 | | 386.15 | | +| | NIM | 1 | 26.91 | 291.97 | 1152.93 | | **H20** | PyTorch | 1 | 37.03 | 381.98 | 1505.14 | | | Diffusers | 1 | | | | +| | NIM | 1 | 67.93 | 711.10 | 2815.02 | | **H100 80GB HBM3** | PyTorch | 1 | 9.15 | 83.30 | | | | Diffusers | 1 | | 234.65 | 830.32 | +| | NIM | 1 | | | | | **H200 141GB HBM3** | PyTorch | 1 | 9.00 | 85.98 | 322.54 | | | Diffusers | 1 | | 231.20 | 825.50 | +| | NIM | 1 | 17.51 | 170.11 | 639.87 | | **B200** | PyTorch | 1 | 5.25 | 45.67 | 168.91 | | | Diffusers | 1 | | 130.78 | 405.87 | +| | NIM | 1 | 9.39 | 94.04 | 346.99 | | **B300** | PyTorch | 1 | 4.92 | 42.14 | 149.55 | | | Diffusers | 1 | | 120.69 | 381.36 | +| | NIM | 1 | 9.29 | 88.67 | 322.94 | | **GB300** | PyTorch | 1 | | | | | | Diffusers | 1 | | 109.65 | 325.76 | +| | NIM | 1 | | | | +| **H100 NVL** | NIM | 1 | 24.54 | 267.92 | 937.49 | +| **H200 NVL** | NIM | 1 | 19.32 | 222.13 | 840.59 | ### Image-to-Video (i2v) @@ -82,18 +93,27 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 15.21 | 154.24 | 587.19 | | | Diffusers | 1 | | | | +| | NIM | 1 | 26.96 | 292.49 | 1157.15 | | **H20** | PyTorch | 1 | 37.09 | 380.01 | 1501.45 | | | Diffusers | 1 | | | | +| | NIM | 1 | 68.26 | 712.78 | 2815.06 | | **H100 80GB HBM3** | PyTorch | 8 | 5.75 | 19.08 | 55.33 | | | Diffusers | 8 | | | | +| | NIM | 8 | | | | | **H200 141GB HBM3** | PyTorch | 1 | 9.02 | 85.61 | 318.65 | | | Diffusers | 1 | | 234.20 | 819.74 | +| | NIM | 1 | 17.53 | 170.44 | 639.64 | | **B200** | PyTorch | 1 | 5.18 | 45.24 | 167.93 | | | Diffusers | 1 | | 129.97 | 408.14 | +| | NIM | 1 | 9.42 | 92.08 | 340.06 | | **B300** | PyTorch | 1 | 4.96 | 41.10 | 151.74 | | | Diffusers | 1 | | 120.90 | | +| | NIM | 1 | 9.27 | 90.00 | 318.53 | | **GB300** | PyTorch | 1 | | | | | | Diffusers | 1 | | 108.89 | 327.89 | +| | NIM | 1 | | | | +| **H100 NVL** | NIM | 1 | 24.59 | 266.77 | 939.10 | +| **H200 NVL** | NIM | 1 | 19.42 | 222.95 | 840.66 | ### Text-to-Image (t2i) @@ -101,21 +121,30 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound |---|---|:-:|---:|---:|---:| | **RTX PRO 6000 Blackwell** | PyTorch | 1 | 1.75 | 2.76 | 5.13 | | | Diffusers | 1 | | | | +| | NIM | 1 | 2.04 | 4.02 | 7.37 | | **H20** | PyTorch | 1 | 2.33 | 5.73 | 12.01 | | | Diffusers | 1 | | | | +| | NIM | 1 | 2.22 | 10.18 | 18.16 | | **H100 80GB HBM3** | PyTorch | 1 | 1.84 | 1.92 | 2.94 | | | Diffusers | 1 | | 21.47 | 24.36 | +| | NIM | 1 | | | | | **H200 141GB HBM3** | PyTorch | 1 | 1.88 | 1.85 | 2.98 | | | Diffusers | 1 | | 19.90 | 23.20 | +| | NIM | 1 | 1.98 | 3.08 | 4.79 | | **B200** | PyTorch | 1 | 1.48 | 1.47 | 1.85 | | | Diffusers | 1 | | 17.38 | 19.21 | +| | NIM | 1 | 1.68 | 1.73 | 2.68 | | **B300** | PyTorch | 1 | 2.37 | 2.29 | 2.47 | | | Diffusers | 1 | | | 18.40 | +| | NIM | 1 | 2.56 | 2.55 | 2.80 | | **GB300** | PyTorch | 1 | | | | | | Diffusers | 1 | | 22.73 | 20.98 | +| | NIM | 1 | | | | +| **H100 NVL** | NIM | 1 | 1.91 | 3.73 | 6.50 | +| **H200 NVL** | NIM | 1 | 1.93 | 3.25 | 5.15 | Notes: -1. All times measured on identical workloads (same seed, sampler settings, prompt). +1. OSS times measured on identical workloads (same seed, sampler settings, prompt). 2. Multi-GPU configurations use tensor parallelism. 3. Values are average generation latency in seconds; lower is better. 4. The **GPUs** column gives the number of GPUs behind every value in that row, and is the same for every engine in a GPU block so the rows can be compared directly. diff --git a/inference_benchmarks/cosmos3-super-generator.md b/inference_benchmarks/cosmos3-super-generator.md index c96c7490..924bebbc 100644 --- a/inference_benchmarks/cosmos3-super-generator.md +++ b/inference_benchmarks/cosmos3-super-generator.md @@ -4,7 +4,7 @@ These tables report **Cosmos3-Super Generator** latency in seconds. Lower is better. -Every row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. Cosmos3-Super is reported at one GPU on H200 NVL, H200 141GB HBM3, B200, and B300, and at eight GPUs on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3. vLLM-Omni does fit on a single GPU on those four smaller-memory parts, but is reported at eight alongside the other runtimes; PyTorch is the engine that sets the eight-GPU configuration on the H200 parts. Because the configuration varies between rows, read the **GPUs** column before comparing latencies down a column. +Each primary vision-generation row reports one GPU configuration, named in the **GPUs** column, and all three resolutions in that row come from it. The configuration is one GPU when the model fits on a single device at its highest resolution, 720p, and eight GPUs when it does not. Within one GPU's block every engine reports the same configuration, so a column can be read straight across the runtimes. Cosmos3-Super is reported at one GPU on H200 NVL, H200 141GB HBM3, B200, and B300, and at eight GPUs on RTX PRO 6000 Blackwell, H20, H100 NVL, and H100 80GB HBM3. vLLM-Omni does fit on a single GPU on those four smaller-memory parts, but is reported at eight alongside the other runtimes; PyTorch is the engine that sets the eight-GPU configuration on the H200 parts. Because the configuration varies between rows, read the **GPUs** column before comparing latencies down a column. Empty cells mean that a run has not been completed for that GPU, engine, or resolution; they do not indicate that a combination is unsupported. @@ -16,6 +16,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso - [Text-to-Video (t2v)](#text-to-video-t2v) - [Image-to-Video (i2v)](#image-to-video-i2v) - [Text-to-Image (t2i)](#text-to-image-t2i) +- [Additional NIM configurations](#additional-nim-configurations) - [Additional audiovisual generation](#additional-audiovisual-generation) - [Video-to-Video (v2v)](#video-to-video-v2v) - [Text-to-Audio-and-Video (t2av)](#text-to-audio-and-video-t2av) @@ -34,7 +35,7 @@ Empty cells mean that a run has not been completed for that GPU, engine, or reso ## Benchmark methodology -The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch and NIM values preserve the previously published campaigns. Those tables use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. +The primary t2v, i2v, and t2i tables cover PyTorch, vLLM-Omni, Diffusers, TensorRT-LLM, and NIM. Their vLLM-Omni values are from PBR `#307999`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (t2i, t2v, i2v)**; the PyTorch values preserve the previously published campaigns. NIM values report average generation time using FP8 latency profiles (see note 7). NIM FP8 output quality has been validated to be comparable to the BF16 baseline. OSS rows use BF16 precision, batch size 1, and matched prompts, seeds, and sampler settings where documented. Video workloads follow the standard Cosmos3 generation profile of 189 frames at 24 FPS unless a resolution tier limits frame count. The additional audiovisual and action tables come from three internal benchmark reports. PyTorch values are from PBR `#308197`, **Cosmos3-Generator OSS Inference Benchmarking 32B and 8B (189 frames)**: average generation (sampling) latency from the native OSS path, using **CUDA Graphs disabled** and the **latency** automatic-sharding preset. vLLM-Omni values for text-to-audio-and-video (`t2av`/`t2vs`) and image-to-audio-and-video (`i2av`/`i2vs`) are from PBR `#308195`. vLLM-Omni action values are from PBR `#308481`, **Cosmos3-Generator vLLM-Omni Inference Benchmarking (action)**, measured with the `vllm/vllm-omni:cosmos3` image and the official action cookbook samples; `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` was not set. Forward-dynamics cells are the mean of `av_forward`, `av_left`, and `av_right`; inverse-dynamics cells are the mean of `av_inverse_0` and `av_inverse_1`. That action sweep covers 1, 2, and 4 GPUs only, because its 8-GPU Ulysses runs failed a sequence-length divisibility check; those rows therefore report the single-GPU configuration. Policy-DROID is a separate checkpoint and was measured only at 480p on one GPU. That action sweep did not cover Super on H20, H100 NVL, or H100 80GB HBM3. Diffusers values come from four further reports: PBR `#308202` (all modalities) and PBR `#308451` (video-to-video) on one GPU, PBR `#308918` for 4- and 8-GPU runs, and PBR `#308587` for transfer. vLLM-Omni transfer values are from PBR `#308574`. Where PBR `#308918` offers several tensor- and context-parallel splits at the same GPU count, the fastest is published. Its 1-GPU numbers are not used, because that sweep ran a different denoising-step budget than the single-GPU reports. TensorRT-LLM values are from PBR `#308000`, **Cosmos3-Generator TRT-LLM Inference Benchmarking**, which sweeps tensor-parallel, CFG-parallel, and Ulysses-parallel splits at each GPU count; the fastest split at the count a row reports is the one published. That campaign covers six GPUs, so the TensorRT-LLM rows for H100 NVL and H200 NVL are empty, and it includes no action workloads. Values are rounded to two decimal places. @@ -66,42 +67,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | — | | | | | | Diffusers | 8 | 25.39 | 154.25 | 437.57 | | | TensorRT-LLM | 8 | 0.21 | 0.13 | 0.10 | -| | NIM | 8 | 13.99 | 99.05 | 286.02 | +| | NIM | 8 | 7.27 | 49.69 | 171.15 | | **H20** | PyTorch | 8 | 27.72 | 152.46 | 492.41 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 26.66 | 161.49 | 517.37 | | | TensorRT-LLM | 8 | 0.06 | 0.01 | 0.01 | -| | NIM | 8 | 12.95 | 110.71 | 395.56 | +| | NIM | 8 | 11.67 | 98.96 | 370.48 | | **H100 NVL** | PyTorch | 8 | 16.83 | 64.14 | 186.19 | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 24.21 | 108.70 | 322.28 | | | TensorRT-LLM | — | | | | -| | NIM | 8 | 12.73 | 66.07 | 197.32 | +| | NIM | 8 | 6.19 | 47.20 | 182.00 | | **H200 NVL** | PyTorch | 8 | 24.78 | 47.69 | 142.35 | | | vLLM-Omni | 1 | 27.54 | 252.33 | 911.49 | | | Diffusers | 1 | 33.00 | 286.80 | 1036.00 | | | TensorRT-LLM | — | | | | -| | NIM | 1 | 17.13 | 200.00 | 811.41 | +| | NIM | 1 | 19.32 | 222.13 | 840.59 | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | — | | | | | | Diffusers | 8 | 19.40 | 56.39 | 174.57 | | | TensorRT-LLM | 8 | 0.04 | 0.02 | 0.04 | -| | NIM | 8 | 5.89 | 35.52 | 114.92 | +| | NIM | 8 | | | | | **H200 141GB HBM3** | PyTorch | 8 | 11.82 | 41.78 | 123.49 | | | vLLM-Omni | 1 | 25.61 | 219.11 | 769.63 | | | Diffusers | 1 | 31.00 | 251.60 | 886.20 | | | TensorRT-LLM | 1 | 24.21 | 234.08 | 853.44 | -| | NIM | 1 | 15.95 | 174.71 | 695.89 | +| | NIM | 1 | 17.51 | 170.11 | 639.87 | | **B200** | PyTorch | 1 | 14.66 | 114.38 | 407.50 | | | vLLM-Omni | 1 | 13.84 | 114.08 | 383.84 | | | Diffusers | 1 | 19.00 | 127.20 | 414.40 | | | TensorRT-LLM | 1 | 12.32 | 104.63 | 367.12 | -| | NIM | 1 | 9.09 | 82.39 | 314.68 | +| | NIM | 1 | 9.39 | 94.04 | 346.99 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 14.57 | 109.03 | 366.66 | | | Diffusers | 1 | 17.73 | 125.70 | 398.94 | | | TensorRT-LLM | 1 | 11.85 | 97.66 | 333.28 | -| | NIM | 1 | 9.67 | 79.73 | 292.35 | +| | NIM | 1 | 9.29 | 88.67 | 322.94 | ### Image-to-Video (i2v) @@ -111,42 +112,42 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | 8 | 11.66 | 80.94 | 240.27 | | | Diffusers | 8 | 26.29 | 156.16 | 540.65 | | | TensorRT-LLM | 8 | 0.19 | 0.13 | 0.09 | -| | NIM | 8 | 14.48 | 100.23 | 289.93 | +| | NIM | 8 | 7.58 | 51.55 | 171.79 | | **H20** | PyTorch | 8 | 28.22 | 153.19 | 491.93 | | | vLLM-Omni | 8 | 19.65 | 148.77 | 488.51 | | | Diffusers | 8 | 27.65 | 165.91 | 530.44 | | | TensorRT-LLM | 8 | 0.06 | 0.01 | 0.01 | -| | NIM | 8 | 14.06 | 114.49 | 405.68 | +| | NIM | 8 | 11.80 | 99.06 | 369.69 | | **H100 NVL** | PyTorch | 8 | 16.96 | 64.17 | 186.47 | | | vLLM-Omni | 8 | 8.48 | 59.44 | 212.53 | | | Diffusers | 8 | 25.45 | 112.36 | 325.56 | | | TensorRT-LLM | — | | | | -| | NIM | 8 | 13.30 | 67.34 | 201.39 | +| | NIM | 8 | 6.23 | 46.21 | 183.17 | | **H200 NVL** | PyTorch | 8 | 24.70 | 47.60 | 141.62 | | | vLLM-Omni | 1 | 27.90 | 254.29 | 915.05 | | | Diffusers | 1 | 33.00 | 287.20 | 1034.60 | | | TensorRT-LLM | — | | | | -| | NIM | 1 | 17.51 | 201.45 | 817.35 | +| | NIM | 1 | 19.42 | 222.95 | 840.66 | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | 8 | 6.48 | 38.31 | | | | Diffusers | 8 | 19.98 | 55.63 | 184.81 | | | TensorRT-LLM | 8 | 0.03 | 0.03 | 0.04 | -| | NIM | 8 | 6.49 | 36.81 | 118.77 | +| | NIM | 8 | | | | | **H200 141GB HBM3** | PyTorch | 8 | 11.80 | 42.10 | 123.57 | | | vLLM-Omni | 1 | 25.47 | 220.70 | 766.33 | | | Diffusers | 1 | 31.00 | 249.20 | 879.20 | | | TensorRT-LLM | 1 | 24.46 | 230.02 | | -| | NIM | 1 | 16.39 | 175.95 | 699.13 | +| | NIM | 1 | 17.53 | 170.44 | 639.64 | | **B200** | PyTorch | 1 | 14.71 | 112.40 | 397.31 | | | vLLM-Omni | 1 | 14.13 | 115.17 | 386.35 | | | Diffusers | 1 | 19.20 | 127.00 | 414.80 | | | TensorRT-LLM | 1 | 12.67 | 105.74 | 372.60 | -| | NIM | 1 | 9.36 | 83.19 | 316.76 | +| | NIM | 1 | 9.42 | 92.08 | 340.06 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 14.14 | 111.42 | 368.73 | | | Diffusers | 1 | 25.21 | 132.23 | 410.81 | | | TensorRT-LLM | 1 | 12.15 | 98.90 | 336.20 | -| | NIM | 1 | 9.73 | 80.51 | 294.11 | +| | NIM | 1 | 9.27 | 90.00 | 318.53 | ### Text-to-Image (t2i) @@ -156,45 +157,119 @@ The PBR uses `t2av`, `v2av`, and `i2av`; some public recipes call the same sound | | vLLM-Omni | 8 | 3.11 | 3.54 | 5.70 | | | Diffusers | 8 | 16.90 | 17.52 | 16.73 | | | TensorRT-LLM | 8 | 0.09 | 0.09 | -0.04 | +| | NIM | 8 | 1.51 | 1.75 | 2.60 | | **H20** | PyTorch | 8 | 14.18 | 16.46 | 20.92 | | | vLLM-Omni | 8 | 5.12 | 5.44 | 8.85 | | | Diffusers | 8 | 18.29 | 18.48 | 18.46 | | | TensorRT-LLM | 8 | 0.02 | 0.13 | 0.03 | +| | NIM | 8 | 1.76 | 5.24 | 3.93 | | **H100 NVL** | PyTorch | 8 | 19.86 | 19.80 | 19.87 | | | vLLM-Omni | 8 | 3.54 | 3.48 | 4.11 | | | Diffusers | 8 | 16.56 | 16.60 | 16.88 | | | TensorRT-LLM | — | | | | +| | NIM | 8 | 1.74 | 4.79 | 2.13 | | **H200 NVL** | PyTorch | 8 | 32.86 | 33.05 | 33.16 | | | vLLM-Omni | 1 | 2.73 | 6.28 | 11.02 | | | Diffusers | 1 | 5.00 | 8.00 | 12.00 | | | TensorRT-LLM | — | | | | +| | NIM | 1 | 1.93 | 3.25 | 5.15 | | **H100 80GB HBM3** | PyTorch | — | | | | | | vLLM-Omni | 8 | 5.09 | 5.16 | 4.91 | | | Diffusers | 8 | 17.97 | 18.03 | 19.28 | | | TensorRT-LLM | 8 | 0.06 | 0.02 | 0.02 | +| | NIM | 8 | | | | | **H200 141GB HBM3** | PyTorch | 8 | 13.48 | 13.53 | 13.50 | | | vLLM-Omni | 1 | 2.83 | 5.70 | 10.24 | | | Diffusers | 1 | 5.00 | 8.00 | 11.00 | | | TensorRT-LLM | 1 | 2.24 | 5.10 | 10.77 | +| | NIM | 1 | 1.98 | 3.08 | 4.79 | | **B200** | PyTorch | 1 | 4.51 | 4.78 | 7.25 | | | vLLM-Omni | 1 | 2.32 | 3.29 | 5.99 | | | Diffusers | 1 | 4.40 | 6.00 | 8.00 | | | TensorRT-LLM | 1 | 1.69 | 2.84 | 5.72 | +| | NIM | 1 | 1.68 | 1.73 | 2.68 | | **B300** | PyTorch | — | | | | | | vLLM-Omni | 1 | 5.05 | 3.79 | 7.08 | | | Diffusers | 1 | 6.49 | 6.76 | 7.11 | | | TensorRT-LLM | 1 | 2.40 | 2.77 | 5.43 | +| | NIM | 1 | 2.56 | 2.55 | 2.80 | Notes: -1. All times measured on identical workloads (same seed, sampler settings, prompt). +1. OSS times measured on identical workloads (same seed, sampler settings, prompt). 2. Multi-GPU configurations use tensor parallelism, except TensorRT-LLM, which may also split across CFG and Ulysses dimensions. 3. vLLM-Omni numbers are for the upcoming public release in the vLLM-Omni repo and are subject to change before GA. 4. Diffusers numbers use the HuggingFace `diffusers` integration without custom CUDA graphs. Single-GPU values come from PBR `#308202` and PBR `#308451`; multi-GPU values from PBR `#308918`. 5. At 256p, multi-GPU configurations on B300 may underperform single-GPU because of small-workload tensor-parallel overhead, so single-GPU is the recommended deployment at this resolution. 6. PyTorch numbers report average generation (sampling) time from OSS inference benchmarking. -7. NIM numbers use latency profiles with FP8 precision and report end-to-end `Request Latency s`, including request processing, video generation, output encoding, and returning the response. +7. NIM numbers use FP8 latency profiles with offload disabled and report `Avg. Generation Time (s)`, excluding request overhead and MP4 encoding. Runs use concurrency 1, three measured requests, and 189 video frames or one image frame. 8. The 720p i2v cell for vLLM-Omni on H100 80GB HBM3 is empty because the multi-GPU runs for it returned times far below the 480p result, which is not physically plausible; the measurement is being rerun. +## Additional NIM configurations + +Average generation time in seconds for GPU counts not shown in the primary tables above. These runs use the same FP8 latency profiles with offload disabled (see note 7). + +| GPU | Modality | GPUs | 256p | 480p | 720p | +|---|---|:-:|---:|---:|---:| +| RTX PRO 6000 Blackwell | t2v | 1 | 26.91 | 291.97 | 1152.93 | +| RTX PRO 6000 Blackwell | t2v | 4 | 10.40 | 89.56 | 319.95 | +| H20 | t2v | 1 | 67.93 | 711.10 | 2815.02 | +| H20 | t2v | 4 | 20.46 | 191.68 | 733.55 | +| H100 NVL | t2v | 1 | 24.54 | 267.92 | 937.49 | +| H100 NVL | t2v | 4 | 8.13 | 106.45 | 384.12 | +| H200 NVL | t2v | 4 | 7.56 | 60.91 | 230.97 | +| H200 NVL | t2v | 8 | 5.00 | 32.38 | 117.52 | +| H100 80GB HBM3 | t2v | 1 | | | | +| H100 80GB HBM3 | t2v | 4 | | | | +| H200 141GB HBM3 | t2v | 4 | 5.66 | 47.23 | 173.93 | +| H200 141GB HBM3 | t2v | 8 | 3.55 | 24.50 | 88.18 | +| B200 | t2v | 4 | 3.49 | 25.14 | 92.65 | +| B200 | t2v | 8 | 2.46 | 13.30 | 46.93 | +| B300 | t2v | 4 | 3.80 | 23.87 | 86.59 | +| B300 | t2v | 8 | 3.58 | 13.50 | 44.95 | +| RTX PRO 6000 Blackwell | i2v | 1 | 26.96 | 292.49 | 1157.15 | +| RTX PRO 6000 Blackwell | i2v | 4 | 10.47 | 89.68 | 320.55 | +| H20 | i2v | 1 | 68.26 | 712.78 | 2815.06 | +| H20 | i2v | 4 | 20.53 | 191.45 | 733.13 | +| H100 NVL | i2v | 1 | 24.59 | 266.77 | 939.10 | +| H100 NVL | i2v | 4 | 8.21 | 105.38 | 386.91 | +| H200 NVL | i2v | 4 | 7.57 | 60.97 | 230.28 | +| H200 NVL | i2v | 8 | 5.14 | 32.61 | 117.43 | +| H100 80GB HBM3 | i2v | 1 | | | | +| H100 80GB HBM3 | i2v | 4 | | | | +| H200 141GB HBM3 | i2v | 4 | 5.76 | 47.14 | 173.77 | +| H200 141GB HBM3 | i2v | 8 | 3.70 | 24.71 | 88.45 | +| B200 | i2v | 4 | 3.53 | 25.12 | 93.12 | +| B200 | i2v | 8 | 2.58 | 13.56 | 46.21 | +| B300 | i2v | 4 | 3.89 | 24.26 | 86.95 | +| B300 | i2v | 8 | 3.68 | 13.73 | 44.87 | +| RTX PRO 6000 Blackwell | t2i | 1 | 2.04 | 4.02 | 7.37 | +| RTX PRO 6000 Blackwell | t2i | 4 | 1.46 | 2.15 | 3.25 | +| H20 | t2i | 1 | 2.22 | 10.18 | 18.16 | +| H20 | t2i | 4 | 1.79 | 3.76 | 6.07 | +| H100 NVL | t2i | 1 | 1.91 | 3.73 | 6.50 | +| H100 NVL | t2i | 4 | 1.61 | 1.71 | 2.46 | +| H200 NVL | t2i | 4 | 1.62 | 1.63 | 2.43 | +| H200 NVL | t2i | 8 | 1.66 | 4.78 | 1.89 | +| H100 80GB HBM3 | t2i | 1 | | | | +| H100 80GB HBM3 | t2i | 4 | | | | +| H200 141GB HBM3 | t2i | 4 | 1.77 | 1.76 | 1.86 | +| H200 141GB HBM3 | t2i | 8 | 1.78 | 5.15 | 1.79 | +| B200 | t2i | 4 | 1.48 | 1.54 | 1.47 | +| B200 | t2i | 8 | 1.60 | 1.49 | 1.54 | +| B300 | t2i | 4 | 2.35 | 2.32 | 2.39 | +| B300 | t2i | 8 | 2.38 | 2.62 | 2.48 | + +Single-GPU H100 SXM 80GB generation times using FP8 latency profiles. Offload mode is shown separately; `unspecified` means it was not set explicitly. + +| GPU | Offload mode | Modality | 256p/1 | 480p/1 | 720p/1 | +|---|---|---|---:|---:|---:| +| H100 80GB HBM3 | model | t2v | 23.98 | 174.92 | 631.25 | +| H100 80GB HBM3 | model | i2v | | | 631.04 | +| H100 80GB HBM3 | model | t2i | 7.98 | | 10.55 | +| H100 80GB HBM3 | unspecified | t2v | 41.47 | 170.37 | 625.43 | +| H100 80GB HBM3 | unspecified | i2v | 41.50 | 170.60 | 625.66 | +| H100 80GB HBM3 | unspecified | t2i | 37.21 | 37.24 | 37.28 | + ## Additional audiovisual generation ### Video-to-Video (v2v)