diff --git a/.gitignore b/.gitignore index 0673d33..d431218 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ __pycache__/ .env runs/ .DS_Store +site/ diff --git a/README.md b/README.md index 7ae0d7a..b11dbdd 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,14 @@ your own ComfyUI. | **Disintegrate** | **Colossus** | **Zero Gravity** | **Frozen Rain** (beta) | | ![](web/thumbs/disintegrate.jpg) | ![](web/thumbs/colossus.jpg) | ![](web/thumbs/zero-gravity.jpg) | ![](web/thumbs/frozen-rain.jpg) | +## Try it + +* **In your browser, no key:** [vfx.nomastudio.ai](https://vfx.nomastudio.ai) plays every effect. + Pick one and open it as a Comfy Cloud app: sign in with your Comfy account, add a photo and run + it. The run bills your own Comfy account. +* **On your machine, everything:** the full app adds video mode, a second photo for the likeness, 9:16 + and HQ. It runs with your own Comfy API key, see [Run it](#run-it). + ## Two ways in **From a photo.** Upload one or two photos of a person. Each preset builds its own film scene @@ -25,11 +33,16 @@ start, and VFX 4 All continues the shot from that exact frame. The result is you cut, then the effect, joined on the same frame so the cut is invisible. Clips from 2 to 60 seconds, 480p to 4K, up to 500 MB (`VIDEO_LIMITS` in `vfx4all/config.py`). +No photo or clip at hand? Each mode loads our own samples with one click (see [samples/](samples/)). +Under **More options**, an extra direction in English goes after the preset's effect prompt, and a +seed repeats a take. + **In Comfy Cloud, no key at all.** Every preset that animates a single first frame (41 of the 45) can also run as a Comfy Cloud App Mode app: one graph that takes your photo, builds the scene and renders the effect. You open it with your own Comfy account and run it with your own credits; nobody hands an API key to anyone. -`tools/build_app_graph.py` builds those graphs from the presets. +`tools/build_app_graph.py` builds those graphs from the presets, and each preset keeps the share link +of its app in `comfy_app`. ## How it works @@ -56,8 +69,8 @@ first frame --> Qwen Image Edit 2511 (or a mirror) --> last frame --> LTX-2.5 fi repeats the uploads and downloads that time out, and splices videos with ffmpeg. * `web/` is the interface: plain HTML, CSS and JavaScript, no build step. It shows the prompt that was sent, keeps your results across reloads and compares a spliced clip with your original. -* `tools/build_app_graph.py` builds the App Mode graphs and `tools/batch_presets.py` renders test - versions of the presets; `tools/dev/` keeps the A/B scripts behind the choices in +* `tools/build_app_graph.py` builds the App Mode graphs, `tools/build_site.py` builds the static + catalog site and `tools/batch_presets.py` renders test versions of the presets; `tools/dev/` keeps the A/B scripts behind the choices in [workflows/README.md](workflows/README.md). ## Run it @@ -76,6 +89,9 @@ cp .env.example .env # then paste your COMFY_API_KEY ffmpeg ships inside the `imageio-ffmpeg` wheel, so there is nothing else to install. `requirements.txt` pins the exact versions the app is tested with. +No photo or clip at hand? `samples/` has two photos and two clips to try both modes; see +[samples/README.md](samples/README.md) for which effect to pair with each. + ### Run the tests ```bash @@ -83,7 +99,7 @@ ffmpeg ships inside the `imageio-ffmpeg` wheel, so there is nothing else to inst ``` The tests run offline in a few seconds. They check every preset and graph, the App Mode builder, -the upload limits, the ffmpeg steps (phone clips shot vertical included), the HTTP API with its +the static site build, the upload limits, the ffmpeg steps (phone clips shot vertical included), the HTTP API with its hosted mode, access code and daily cap, the job queue, and that every model the graphs load is listed in `THIRD_PARTY_NOTICES.md`. They never reach Comfy Cloud (the test package blanks `COMFY_API_KEY` before the app loads), so they cost nothing. GitHub Actions runs them on Linux and @@ -102,8 +118,14 @@ Every generation bills the Comfy Cloud workspace that owns the API key it runs w uses it for that job only and never stores it. If they leave it empty, the server key from `.env` is used, so read [Hosting it](#hosting-it) before you expose the app. * **Your own GPU, no cloud credits:** the models are open weights, so the same app can drive your own - ComfyUI through [comfy-api-proxy](https://github.com/Comfy-Org/comfy-api-proxy). Set + ComfyUI through [comfy-api-proxy](https://github.com/Comfy-Org/comfy-api-proxy), with no key. Set `COMFY_BASE_URL=http://127.0.0.1:8189` in `.env`. +* **A Comfy deployment:** a ComfyUI of your own on a dedicated cloud GPU, billed while it is up. Set + `COMFY_BASE_URL` to its URL. + +[docs/RUN_ON_YOUR_COMFYUI.md](docs/RUN_ON_YOUR_COMFYUI.md) lists every model file the graphs load, with +its source, revision and download command, and the steps for the proxy and for a deployment. The top +bar of the app says where it runs. Only open-weights models run in the workflows, so no partner API credits are used: Comfy Cloud charges GPU time. In our tests on Comfy Cloud a scene took about 10 seconds of GPU and a 6 second @@ -235,6 +257,22 @@ sends a strict Content Security Policy and serves the fonts itself, so the page other sites. If you host it for others, the LTX license asks you to put its Section 4 and the Acceptable Use Policy into your own terms (see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)). +### The catalog site + +A public page that spends nothing does not need the server at all. `tools/build_site.py` writes +`site/`: the same interface in static mode, where every card plays its preview and opens the +preset's Comfy Cloud app (`comfy_app`), so each visitor runs it with their own Comfy account. The +studio and the key settings are left out, `presets.json` replaces the API, and `vercel.json` sends +the same security headers as the server. + +```bash +.venv/Scripts/python tools/build_site.py --demo path/to/demo.mp4 # --demo is optional +cd site && vercel deploy # or any static host +``` + +On Vercel, the first deployment of a project goes to production and takes the project's domains, +even without `--prod`. + ## Consent Only use photos and videos of people who agreed to it. The app asks you to confirm that before diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 358773b..9d881ff 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -116,6 +116,7 @@ not licensed under MIT. ## Demo media The thumbnails in `web/thumbs/`, the preview loops in `web/previews/` and the hero image and video in -`web/media/` were made with VFX 4 All from photos of members of the NoMa Studio AI team. They show -real people, so they are not licensed under MIT: do not reuse them outside this project. Use your -own media when you fork it. +`web/media/` were made with VFX 4 All from photos of members of the NoMa Studio AI team, and the +photos and clips in `samples/` show the same people. They show real people who agreed to this use, +so they are not licensed under MIT: do not reuse them outside this project. Use your own media when +you fork it. diff --git a/docs/RUN_ON_YOUR_COMFYUI.md b/docs/RUN_ON_YOUR_COMFYUI.md new file mode 100644 index 0000000..3db932a --- /dev/null +++ b/docs/RUN_ON_YOUR_COMFYUI.md @@ -0,0 +1,142 @@ +# Run VFX 4 All on your own ComfyUI or on a Comfy deployment + +VFX 4 All talks to the Comfy API v2 through `comfy-sdk`, so three places can run its graphs: + +| Where | `COMFY_BASE_URL` in `.env` | `COMFY_API_KEY` | Who pays | +|---|---|---|---| +| Comfy Cloud (the default) | leave it out | your Comfy API key | your Comfy workspace, for the GPU time | +| Your own ComfyUI, through [comfy-api-proxy](https://github.com/Comfy-Org/comfy-api-proxy) | `http://127.0.0.1:8189` | leave it empty | nobody: it runs on your GPU | +| A Comfy Developer Platform deployment | `https://.run.comfy.app` | the key of the workspace that owns the deployment | that workspace, while the deployment is up | + +The top bar of the app says where it runs: "Comfy Cloud ready", "Your ComfyUI ready" or "Your Comfy +deployment ready". Every model is open weights, so the results are the same graphs wherever they run. + +## What the graphs need + +**ComfyUI:** a recent release. Every node the graphs use ships with ComfyUI itself, the LTX-2.5 +keyframe nodes (`LTXV Add Generated Keyframes` and the rest) included, so no custom node pack is +needed. If your ComfyUI is older than the graphs, a job fails with the name of the missing node. + +**Model files:** the graphs load them by the names below, so each file must sit in that folder of +`ComfyUI/models/` under exactly that name. + +| File | Folder | Used by | Source (repository, file, revision) | +|---|---|---|---| +| `qwen_image_edit_2511_int8_convrot.safetensors` | `diffusion_models/` | scenes, last frames | `Comfy-Org/Qwen-Image-Edit_ComfyUI`, `split_files/diffusion_models/...`, `f68ace85e60b4a02a323e394253731947657b7d2` | +| `qwen_2.5_vl_7b_fp8_scaled.safetensors` | `text_encoders/` | scenes, last frames | `Comfy-Org/Qwen-Image_ComfyUI`, `split_files/text_encoders/...`, `1f12b17be14c89b026c51a91d67c32f84bb047bc` | +| `qwen_image_vae.safetensors` | `vae/` | scenes, last frames | `Comfy-Org/Qwen-Image_ComfyUI`, `split_files/vae/...`, `1f12b17be14c89b026c51a91d67c32f84bb047bc` | +| `ltx-2.5-22b-distilled-transformer-comfy-int8-convrot.safetensors` | `diffusion_models/` | every effect | `Lightricks/LTX-2.5`, `diffusion_models/...`, `49b093a39c19169a8a81721dcc541e07e945232f` | +| `gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors` | `text_encoders/` | every effect | `Lightricks/LTX-2.5`, `text_encoders/...`, same revision | +| `ltx-2.5-video-vae-bf16.safetensors` | `vae/` | every effect | `Lightricks/LTX-2.5`, `vae/...`, same revision | +| `ltx-2.5-audio-vae-bf16.safetensors` | `vae/` | every effect | `Lightricks/LTX-2.5`, `vae/...`, same revision | +| `ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors` | `latent_upscale_models/` | every effect | `Lightricks/LTX-2.5`, `latent_upscale_models/...`, same revision | +| `ltx-2.5-latent-temporal-upscaler-x2-bf16-1.0.safetensors` | `latent_upscale_models/` | every effect | `Lightricks/LTX-2.5`, `latent_upscale_models/...`, same revision | +| `ltx-2.5-22b-ic-lora-refine-details-1.0.safetensors` | `loras/` | every effect | `Lightricks/LTX-2.5-22b-IC-LoRA-Refine-Details`, at the root, `4912c478b35dd96b6cdcc5f7c7ff9d72c2c57929` | +| `ltx-2.5-22b-distilled-transformer-bf16.safetensors` | `diffusion_models/` | only `VFX4ALL_EFFECT_PIPELINE=classic` | `Lightricks/LTX-2.5`, `diffusion_models/...`, same revision | + +Five presets add a style LoRA. Their files have other names on Hugging Face, so they are renamed when +you download them: + +| File the graph loads | Preset | Source (repository, file, revision) | +|---|---|---| +| `LTX_2.3_Cinematic_Sci-fi-Cyberpunk_Style.safetensors` | Hologram | `vrgamedevgirl84/LTX_2.3_Cinematic_Sci-fi-Cyberpunk_Style_LoRa`, `Cinematic_sci-fi-cyberpunk.safetensors`, `2d2e1796710c62882cba8cea4e51f5100e352c24` | +| `LTX_2.3_Clay_Mation_Style.safetensors` | Claymation | `vrgamedevgirl84/LTX_2.3_Clay_Mation_Style_LoRa`, `Claymation.safetensors`, `bbf7937db81244e7855adadc459ec7f1813b34ac` | +| `LTX2.3_Cozy_Felt_Style.safetensors` | Stop Motion | `vrgamedevgirl84/LTX2.3_Cozy_Felt_Style_LoRa`, `CozyFelt.safetensors`, `3a0b5ec60a63f039c24fc18daa388031a1ee30f4` | +| `LTX_2.3_Fantasy_Anime_Style.safetensors` | Anime Shift | `vrgamedevgirl84/LTX_2.3_Fantasy_Anime_Style_LoRa`, `Fantasy_Anime.safetensors`, `b4853b426fb37c7f4f0c28bf477c98a25aafeba3` | +| `LTX_2.3_Fantasy_Painterly_Style.safetensors` | Paint World | `vrgamedevgirl84/LTX_2.3_Fantasy_Painterly_Style_LoRa`, `Fantasy_Painterly.safetensors`, `a921cc65c53e3b46507165968801a1994f4f4803` | + +The two LTX repositories are gated: accept their terms on Hugging Face and run `hf auth login` once. +Read the licenses in [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md) before you download, the +style LoRAs' in particular. + +### Download them + +Run this from the ComfyUI folder (the one that holds `models/`). The Comfy-Org repositories keep +their files under `split_files/`, so those land in a scratch folder first and move into place. + +```bash +python -m pip install -U huggingface_hub +hf auth login + +# LTX-2.5: the repository's folders match ComfyUI's +LTX=49b093a39c19169a8a81721dcc541e07e945232f +for f in diffusion_models/ltx-2.5-22b-distilled-transformer-comfy-int8-convrot.safetensors \ + text_encoders/gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors \ + vae/ltx-2.5-video-vae-bf16.safetensors vae/ltx-2.5-audio-vae-bf16.safetensors \ + latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \ + latent_upscale_models/ltx-2.5-latent-temporal-upscaler-x2-bf16-1.0.safetensors; do + hf download Lightricks/LTX-2.5 "$f" --revision $LTX --local-dir models +done +hf download Lightricks/LTX-2.5-22b-IC-LoRA-Refine-Details ltx-2.5-22b-ic-lora-refine-details-1.0.safetensors \ + --revision 4912c478b35dd96b6cdcc5f7c7ff9d72c2c57929 --local-dir models/loras + +# Qwen Image Edit 2511 and its text encoder and VAE +hf download Comfy-Org/Qwen-Image-Edit_ComfyUI split_files/diffusion_models/qwen_image_edit_2511_int8_convrot.safetensors \ + --revision f68ace85e60b4a02a323e394253731947657b7d2 --local-dir hf_tmp +hf download Comfy-Org/Qwen-Image_ComfyUI split_files/text_encoders/qwen_2.5_vl_7b_fp8_scaled.safetensors \ + split_files/vae/qwen_image_vae.safetensors --revision 1f12b17be14c89b026c51a91d67c32f84bb047bc --local-dir hf_tmp +mv hf_tmp/split_files/diffusion_models/* models/diffusion_models/ +mv hf_tmp/split_files/text_encoders/* models/text_encoders/ +mv hf_tmp/split_files/vae/* models/vae/ + +# The five style LoRAs, under the names the graphs load +hf download vrgamedevgirl84/LTX_2.3_Cinematic_Sci-fi-Cyberpunk_Style_LoRa Cinematic_sci-fi-cyberpunk.safetensors --revision 2d2e1796710c62882cba8cea4e51f5100e352c24 --local-dir hf_tmp +hf download vrgamedevgirl84/LTX_2.3_Clay_Mation_Style_LoRa Claymation.safetensors --revision bbf7937db81244e7855adadc459ec7f1813b34ac --local-dir hf_tmp +hf download vrgamedevgirl84/LTX2.3_Cozy_Felt_Style_LoRa CozyFelt.safetensors --revision 3a0b5ec60a63f039c24fc18daa388031a1ee30f4 --local-dir hf_tmp +hf download vrgamedevgirl84/LTX_2.3_Fantasy_Anime_Style_LoRa Fantasy_Anime.safetensors --revision b4853b426fb37c7f4f0c28bf477c98a25aafeba3 --local-dir hf_tmp +hf download vrgamedevgirl84/LTX_2.3_Fantasy_Painterly_Style_LoRa Fantasy_Painterly.safetensors --revision a921cc65c53e3b46507165968801a1994f4f4803 --local-dir hf_tmp +mv hf_tmp/Cinematic_sci-fi-cyberpunk.safetensors models/loras/LTX_2.3_Cinematic_Sci-fi-Cyberpunk_Style.safetensors +mv hf_tmp/Claymation.safetensors models/loras/LTX_2.3_Clay_Mation_Style.safetensors +mv hf_tmp/CozyFelt.safetensors models/loras/LTX2.3_Cozy_Felt_Style.safetensors +mv hf_tmp/Fantasy_Anime.safetensors models/loras/LTX_2.3_Fantasy_Anime_Style.safetensors +mv hf_tmp/Fantasy_Painterly.safetensors models/loras/LTX_2.3_Fantasy_Painterly_Style.safetensors +``` + +LTX-2.5 is a 22B model. We render on Comfy Cloud and have not measured how much GPU memory it needs on +one consumer card; start with Draft. + +## Your own ComfyUI, through comfy-api-proxy + +The proxy puts the Comfy API v2 in front of a ComfyUI you run, so the app needs no change. + +1. Start ComfyUI as usual, on `127.0.0.1:8188`. +2. Start the proxy. `--max-upload-mb 500` matches the app's own limit for clips: + + ```bash + pip install comfy-api-proxy + comfy-api-proxy --comfyui http://127.0.0.1:8188 --port 8189 --max-upload-mb 500 + ``` + +3. In `.env`, set `COMFY_BASE_URL=http://127.0.0.1:8189` and leave `COMFY_API_KEY` empty. The proxy + asks for no key while it stays on `127.0.0.1` without `--token`. +4. Start the app with `python run.py`. The top bar says "Your ComfyUI ready". + +The first job loads the models into memory and takes longer than the ones after it. We have run the +app end to end on Comfy Cloud; this path follows the same API contract through the proxy, and we have +not yet timed it on a local GPU. + +## A Comfy Developer Platform deployment + +A deployment is a ComfyUI of your own on a dedicated cloud GPU. It also runs models and LoRAs that are +not in the Comfy Cloud catalog. Prepare a ComfyUI folder with the files above, then package it with the +Comfy CLI and publish a release: + +```bash +pip install comfy-cli +comfy setup --where cloud +cd /path/to/ComfyUI +comfy build init --name vfx4all --models-dir ./models --custom-nodes-dir ./custom_nodes +comfy build status +comfy build push --release --target linux/nvidia +comfy deploy refs compute +``` + +Create a deployment from that release on the Developer Platform, then set `COMFY_BASE_URL` to its URL +and `COMFY_API_KEY` to the key of the workspace that owns it. A deployment bills for as long as it is +up, so stop or delete it when you finish. + +## When a job says something is missing + +The app reads ComfyUI's own validation error and names what is missing: "This Comfy server has no +X node" (update ComfyUI) or "This Comfy server does not have the model file X" (download it as above, +into the folder the tables give, then restart ComfyUI). diff --git a/presets/walk-of-fire.json b/presets/walk-of-fire.json index d49d3c4..3e9701b 100644 --- a/presets/walk-of-fire.json +++ b/presets/walk-of-fire.json @@ -19,5 +19,5 @@ "scene_prompt": "Create a new cinematic film still in {aspect_words} format. {references} The {subject} is the only person in the new still, with exactly {their} face, eyes, nose, jaw, hairline, hairstyle, hair colour{facial_hair}, skin tone and build{identity_notes}. The clothing, the place and the light come only from this description. The {subject} walks straight toward the camera down the middle of an empty industrial street at night, mid-stride, wearing {wardrobe}, hands loose at the sides. Red-brick warehouses line both sides, sodium streetlights cast pools of orange light, steam rises from a grate in the distance, and the wet asphalt holds long glossy reflections. Low-angle wide shot from knee height, symmetrical one-point perspective with the street vanishing behind {them}, the full body filling the middle third of the frame height. Cinematic night photography, anamorphic look, deep shadows with warm sodium highlights and a cool blue moonlight rim on the shoulders. Correct human anatomy, natural skin texture, sharp focus on the {subject}, plain unmarked brick walls throughout.", "effect_prompt": "Use the provided start image as the first frame. The {subject} is ALREADY walking toward the camera in a calm, heavy, unhurried \"funeral march\" stride, weight landing on each heel, shoulders level, arms swinging loose, and keeps walking at exactly the same pace for the whole shot. On the second step, fire erupts from {their} shoulders and spreads over the whole body in a single roaring burst, and the {subject} walks on fully engulfed in tall orange flames, the fire streaming backward like a cape, while {their} face stays clear, calm and unharmed inside the blaze. Every footstep leaves a burning footprint on the wet asphalt, and the flames throw dancing orange light across the brick walls and long fire reflections across the puddles. {Their} eyes stay locked on a point just past the camera, one slow deliberate blink when the flames reach the collar, and the stride never breaks. The camera holds a low angle and slowly dollies backward at walking pace, keeping the {subject} centered and the same size in frame. Sound: a deep whoomph as the fire ignites, then a steady roaring blaze, heavy footsteps on wet asphalt and a low distant city hum. Photorealistic, cinematic night visual effects, fire with real weight and turbulence, sharp detail, stable picture.", "generic_prompt": "Use the provided start image as the first frame. The {subject} in the frame carries on with exactly the same action and pace. Within the first second, fire erupts from {their} shoulders and spreads over {their} whole body in a single roaring burst, tall orange flames streaming upward and backward with real weight and turbulence, while {their} face stays clear, calm and unharmed inside the blaze. Embers drift upward, dark smoke rises, and the firelight throws flickering orange light across everything around {them}. The camera keeps the same framing and the same movement it already has. Sound: a deep whoomph as the fire ignites, then a steady roaring blaze and crackling embers. Photorealistic, cinematic visual effects, sharp detail, stable picture.", - "comfy_app": "" + "comfy_app": "https://cloud.comfy.org/?share=21dcb7ff424d" } diff --git a/samples/README.md b/samples/README.md new file mode 100644 index 0000000..94ae95f --- /dev/null +++ b/samples/README.md @@ -0,0 +1,13 @@ +# Samples + +Files to try VFX 4 All without your own material. They show members of the NoMa Studio AI team who +agreed to this use, so they are **not** covered by the MIT license: use them only to test VFX 4 All, +and do not reuse them elsewhere. + +| File | What it is | Try it with | +|---|---|---| +| `photos/front.jpg` and `photos/side.jpg` | Two photos of the same person, a front view and a side view, plain background | Photo mode, both photos at once, subject "Man" | +| `clips/street-walk-16x9.mp4` | 6 s, 896x512, a man walking toward the camera on a city avenue at sunrise, with sound. Generated with VFX 4 All's own LTX-2.5 graph from a photo scene, to show that AI clips work too | Video mode, cut at about 1.5 s, Walk of Fire, Disintegrate or Portal Jump | +| `clips/rooftop-walk-vertical.mp4` | 3.4 s, 720x1280, a woman walking toward a phone camera on a rooftop, no sound. Real phone footage | Video mode, cut at about 1 s, subject "Woman", Shockwave or Lightning Strike | + +The photos have their metadata removed. diff --git a/samples/clips/rooftop-walk-vertical.mp4 b/samples/clips/rooftop-walk-vertical.mp4 new file mode 100644 index 0000000..f927507 Binary files /dev/null and b/samples/clips/rooftop-walk-vertical.mp4 differ diff --git a/samples/clips/street-walk-16x9.mp4 b/samples/clips/street-walk-16x9.mp4 new file mode 100644 index 0000000..f86c294 Binary files /dev/null and b/samples/clips/street-walk-16x9.mp4 differ diff --git a/samples/photos/front.jpg b/samples/photos/front.jpg new file mode 100644 index 0000000..5236b32 Binary files /dev/null and b/samples/photos/front.jpg differ diff --git a/samples/photos/side.jpg b/samples/photos/side.jpg new file mode 100644 index 0000000..bff4284 Binary files /dev/null and b/samples/photos/side.jpg differ diff --git a/tests/test_config_pipeline.py b/tests/test_config_pipeline.py index 05d703e..f385bc7 100644 --- a/tests/test_config_pipeline.py +++ b/tests/test_config_pipeline.py @@ -8,13 +8,13 @@ from vfx4all import media from vfx4all.config import (HOSTED_VIDEO_LIMITS, IMAGE_LIMITS, VIDEO_LIMITS, check_image, check_video, - closest_selector, video_limits) -from vfx4all.engine import Engine + closest_selector, comfy_target, video_limits) +from vfx4all.engine import Engine, MissingApiKey from vfx4all.jobs import Job -from vfx4all.pipeline import flf_size, keyframes, run_scene, supports -from vfx4all.presets import get_preset +from vfx4all.pipeline import flf_size, keyframes, run_scene, run_splice, supports +from vfx4all.presets import MAX_DIRECTION, get_preset, with_direction -from .helpers import make_image +from .helpers import make_clip, make_image MB = 1024 * 1024 @@ -97,6 +97,52 @@ def fake_scene(engine, refs, prompt, size, seed, dest, progress): self.assertIn("Image 1 and image 2 are two photos of the same man.", sent[2]) +class TargetTests(unittest.TestCase): + def test_the_base_url_says_where_the_graphs_run(self): + cases = {"": "cloud", "https://cloud.comfy.org": "cloud", "https://my-app.run.comfy.app": "deployment", + "http://127.0.0.1:8189": "local", "http://localhost:8189": "local", + "https://gpu.example.com": "custom"} + for url, kind in cases.items(): + with self.subTest(url=url), mock.patch.dict(os.environ, {"COMFY_BASE_URL": url}): + self.assertEqual(comfy_target()["kind"], kind) + + def test_your_own_comfyui_runs_without_a_key(self): + with mock.patch.dict(os.environ, {"COMFY_BASE_URL": "http://127.0.0.1:8189", "COMFY_API_KEY": ""}), \ + mock.patch("vfx4all.engine.Comfy") as comfy: + Engine().client + comfy.assert_called_once_with(api_key=None) + with mock.patch.dict(os.environ, {"COMFY_BASE_URL": "", "COMFY_API_KEY": ""}): + with self.assertRaises(MissingApiKey): + Engine().client + + +class DirectionTests(unittest.TestCase): + def test_the_direction_ends_the_effect_prompt(self): + self.assertEqual(with_direction("A burst of fire.", " "), "A burst of fire.") + self.assertEqual(with_direction("A burst of fire.", " the camera pushes in "), + "A burst of fire. the camera pushes in.") + self.assertEqual(with_direction("Fire.", "Slow motion!"), "Fire. Slow motion!") + self.assertEqual(len(with_direction("Fire.", "a" * 900)), len("Fire. ") + MAX_DIRECTION + 1) + + def test_a_splice_sends_the_direction_to_ltx(self): + sent = {} + + def fake_effect(engine, image, prompt, selector, megapixels, duration, seed, dest, progress, + lora=None, pipeline=None): + sent["prompt"] = prompt + return make_clip(dest.with_suffix(".mp4"), 3.0) # long enough for the poster 1 s after the cut + + with mock.patch.object(Engine, "effect", fake_effect), tempfile.TemporaryDirectory() as tmp: + clip = make_clip(Path(tmp) / "walk.mp4", 2.0) + job = Job(id="splice-direction", kind="splice") + job.dir.mkdir(parents=True, exist_ok=True) + run_splice(job, key=None, preset_id="disintegrate", video=clip, seconds=1.0, subject="man", + quality="draft", audio_mode="effect", seed=7, direction="the wind blows from the left") + self.assertTrue(sent["prompt"].endswith(" the wind blows from the left.")) + self.assertEqual(job.params["prompt"], sent["prompt"]) + self.assertIn("video", job.outputs) + + class KeyframeTests(unittest.TestCase): def test_an_ordinary_preset_needs_no_keyframes(self): # engine=None: if keyframes() tried to call Comfy, this would raise. diff --git a/tests/test_jobs.py b/tests/test_jobs.py index 05a3658..89156b0 100644 --- a/tests/test_jobs.py +++ b/tests/test_jobs.py @@ -10,7 +10,7 @@ from unittest import mock import httpx -from comfy_sdk.exceptions import JobFailed +from comfy_sdk.exceptions import InvalidWorkflow, JobFailed from vfx4all.config import RUNS_DIR, UPLOADS_DIR from vfx4all.jobs import Cancelled, JobManager, LimitReached, Usage, friendly_error @@ -29,6 +29,27 @@ def test_each_failure_reads_as_a_sentence(self): with self.subTest(phrase=phrase): self.assertIn(phrase, friendly_error(exc, "job-1")) + def test_a_missing_node_or_model_is_named(self): + # how ComfyUI words them, directly or behind comfy-api-proxy + node = InvalidWorkflow("Cannot execute because node LTXVAddGeneratedKeyframes does not exist.", + code="invalid_workflow") + model = InvalidWorkflow("Prompt outputs failed validation", code="invalid_workflow", details={ + "node_errors": {"4": {"errors": [{"type": "value_not_in_list", "message": "Value not in list", + "details": "unet_name: 'qwen_image_edit_2511_int8_convrot.safetensors' not in []"}]}}}) + self.assertIn("no LTXVAddGeneratedKeyframes node", friendly_error(node, "job-1")) + said = friendly_error(model, "job-1") + self.assertIn("model file qwen_image_edit_2511_int8_convrot.safetensors", said) + self.assertIn("docs/RUN_ON_YOUR_COMFYUI.md", said) + other = InvalidWorkflow("Prompt outputs failed validation", code="invalid_workflow") + self.assertIn("renamed in its catalog", friendly_error(other, "job-1")) + + def test_the_messages_name_where_the_graph_runs(self): + with mock.patch.dict(os.environ, {"COMFY_BASE_URL": "http://127.0.0.1:8189"}): + down = friendly_error(httpx.ConnectError("refused"), "job-1") + self.assertIn("Check that ComfyUI and comfy-api-proxy are running", down) + self.assertIn("The render failed on your ComfyUI", friendly_error(JobFailed("job failed"), "job-1")) + self.assertIn("The render failed on Comfy Cloud", friendly_error(JobFailed("job failed"), "job-1")) + class UsageTests(unittest.TestCase): def test_the_cap_holds_across_instances(self): diff --git a/tests/test_server.py b/tests/test_server.py index d5913c3..b324eea 100644 --- a/tests/test_server.py +++ b/tests/test_server.py @@ -56,6 +56,45 @@ def test_health_reports_the_limits_and_the_mode(self): self.assertFalse(body["access_required"]) self.assertEqual(body["video_limits"], VIDEO_LIMITS) self.assertEqual(body["image_limits"], IMAGE_LIMITS) + self.assertEqual(body["target"], {"kind": "cloud", "label": "Comfy Cloud"}) + + def test_your_own_comfyui_needs_no_key(self): + with mock.patch.dict(os.environ, {"COMFY_BASE_URL": "http://127.0.0.1:8189"}): + body = self.client.get("/api/health").json() + self.assertTrue(body["has_key"]) + self.assertEqual(body["target"], {"kind": "local", "label": "Your ComfyUI"}) + + def test_a_job_for_your_own_comfyui_is_taken_without_a_key(self): + upload = self.upload(make_clip(self.dir / "own-comfyui.mp4", 3)).json() + body = {"preset": "disintegrate", "video": upload["id"], "seconds": 0.5, "seed": 7, "direction": "slow wind"} + self.assertEqual(self.client.post("/api/jobs/splice", json=body).status_code, 401) + with mock.patch.dict(os.environ, {"COMFY_BASE_URL": "http://127.0.0.1:8189"}), \ + mock.patch.object(server.pipeline, "run_splice") as run: + res = self.client.post("/api/jobs/splice", json=body) + self.assertEqual(res.status_code, 200, res.text) + job = res.json() + for _ in range(50): + if run.called: + break + time.sleep(0.05) + self.assertEqual(job["params"]["direction"], "slow wind") + self.assertEqual(job["params"]["seed"], 7) + self.assertEqual(run.call_args.kwargs["direction"], "slow wind") + + def test_the_samples_are_served_and_nothing_else(self): + photo = self.client.get("/samples/photos/front.jpg") + self.assertEqual(photo.status_code, 200) + self.assertEqual(photo.headers["content-type"], "image/jpeg") + self.assertEqual(self.client.get("/samples/clips/street-walk-16x9.mp4").status_code, 200) + for path in ("/samples/photos/../README.md", "/samples/photos/%2E%2E%2FREADME.md", "/samples/runs/front.jpg", + "/samples/photos/missing.jpg"): + with self.subTest(path=path): + self.assertIn(self.client.get(path).status_code, (404, 422)) + + def test_a_long_direction_is_refused(self): + res = self.client.post("/api/jobs/effect", json={"preset": "walk-of-fire", "image": "x" * 32, + "direction": "a" * 501}) + self.assertEqual(res.status_code, 422) def test_presets(self): cards = self.client.get("/api/presets").json() diff --git a/tests/test_site.py b/tests/test_site.py new file mode 100644 index 0000000..5d4c7d9 --- /dev/null +++ b/tests/test_site.py @@ -0,0 +1,65 @@ +"""The static catalog site builds from web/ and the presets, with no server behind it.""" + +import json +import sys +import tempfile +import unittest +from pathlib import Path + +from vfx4all.config import ROOT +from vfx4all.presets import load_presets + +sys.path.insert(0, str(ROOT / "tools")) +import build_site # noqa: E402 + + +class SiteTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.tmp = tempfile.TemporaryDirectory() + cls.out = Path(cls.tmp.name) / "site" + demo = Path(cls.tmp.name) / "demo.mp4" + demo.write_bytes(b"not really a video") + cls.report = build_site.build(cls.out, demo) + cls.html = (cls.out / "index.html").read_text(encoding="utf-8") + + @classmethod + def tearDownClass(cls): + cls.tmp.cleanup() + + def test_page_runs_in_static_mode(self): + self.assertIn(build_site.STATIC_MODE, self.html) + self.assertNotIn(build_site.APP_MODE, self.html) + self.assertNotIn("{{v}}", self.html) + self.assertTrue((self.out / "static" / "app.js").exists()) + + def test_every_preset_is_listed_with_its_app_mode_status(self): + presets = json.loads((self.out / "static" / "presets.json").read_text(encoding="utf-8")) + self.assertEqual(len(presets), len(load_presets())) + apps = {p.stem for p in (ROOT / "workflows" / "apps").glob("*.json")} + for preset in presets: + self.assertEqual(preset["app_mode"], preset["id"] in apps, preset["id"]) + self.assertTrue((self.out / preset["thumb"].lstrip("/")).exists(), preset["thumb"]) + self.assertTrue((self.out / preset["preview"].lstrip("/")).exists(), preset["preview"]) + self.assertEqual(self.report["app_mode"], len(apps)) + + def test_app_links_are_public_share_links(self): + # a # record link only opens for members of the workspace + for preset in load_presets(): + link = preset.get("comfy_app") or "" + if link: + self.assertTrue(link.startswith("https://cloud.comfy.org/?share="), preset["id"]) + + def test_demo_replaces_the_marker(self): + self.assertIn(build_site.DEMO_VIDEO, self.html) + self.assertTrue((self.out / "static" / "media" / "demo.mp4").exists()) + + def test_headers_match_the_server(self): + from vfx4all.server import CSP + vercel = json.loads((self.out / "vercel.json").read_text(encoding="utf-8")) + sent = {h["key"]: h["value"] for h in vercel["headers"][0]["headers"]} + self.assertEqual(sent["Content-Security-Policy"], CSP) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/build_site.py b/tools/build_site.py new file mode 100644 index 0000000..d7d103e --- /dev/null +++ b/tools/build_site.py @@ -0,0 +1,99 @@ +"""Build the static catalog site: the web interface without the server. Every effect shows its preview +and links to its Comfy Cloud App Mode app (the preset's `comfy_app`), so visitors run it with their own +Comfy account. The output goes to Vercel or any static host. + + python tools/build_site.py # writes site/ + python tools/build_site.py --demo path/to/demo.mp4 # with the demo video +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import shutil +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from vfx4all import __version__ # noqa: E402 +from vfx4all.config import WORKFLOWS_DIR # noqa: E402 +from vfx4all.presets import load_presets, public_view # noqa: E402 + +WEB = ROOT / "web" +APP_MODE = '' +STATIC_MODE = '' +DEMO_MARKER = "" +DEMO_VIDEO = '' + +# The same headers the server sends (vfx4all/server.py), so the site runs under the same rules. +CSP = ("default-src 'self'; script-src 'self'; style-src 'self'; font-src 'self'; img-src 'self' data: blob:; " + "media-src 'self' blob:; connect-src 'self'; object-src 'none'; base-uri 'self'; form-action 'self'; " + "frame-ancestors 'none'") +VERCEL = { + "cleanUrls": True, + "headers": [ + {"source": "/(.*)", "headers": [ + {"key": "Content-Security-Policy", "value": CSP}, + {"key": "X-Content-Type-Options", "value": "nosniff"}, + {"key": "Referrer-Policy", "value": "strict-origin-when-cross-origin"}, + {"key": "Permissions-Policy", "value": "camera=(), microphone=(), geolocation=()"}, + ]}, + {"source": "/static/(.*)", "headers": [{"key": "Cache-Control", "value": "public, max-age=3600"}]}, + ], +} + + +def asset_version() -> str: + """Same cache-busting tag as the server: the version plus the bytes of app.js and app.css.""" + digest = hashlib.sha256(__version__.encode()) + for name in ("app.js", "app.css"): + digest.update((WEB / name).read_bytes()) + return digest.hexdigest()[:10] + + +def site_presets() -> list[dict]: + """The cards, plus whether the preset can run as a single App Mode graph at all.""" + out = [] + for preset in load_presets(): + view = public_view(preset) + view["app_mode"] = (WORKFLOWS_DIR / "apps" / f"{preset['id']}.json").exists() + out.append(view) + return out + + +def build(out: Path, demo: Path | None = None) -> dict: + if out.exists(): + shutil.rmtree(out) + shutil.copytree(WEB, out / "static", ignore=shutil.ignore_patterns("index.html")) + + html = (WEB / "index.html").read_text(encoding="utf-8") + if APP_MODE not in html: + raise RuntimeError("web/index.html lost its vfx4all-mode meta tag") + html = html.replace(APP_MODE, STATIC_MODE).replace("{{v}}", asset_version()) + if demo: + shutil.copyfile(demo, out / "static" / "media" / "demo.mp4") + html = html.replace(DEMO_MARKER, DEMO_VIDEO) + (out / "index.html").write_text(html, encoding="utf-8") + + presets = site_presets() + (out / "static" / "presets.json").write_text(json.dumps(presets, ensure_ascii=False), encoding="utf-8") + (out / "vercel.json").write_text(json.dumps(VERCEL, indent=2), encoding="utf-8") + return {"presets": len(presets), "apps": sum(bool(p["comfy_app"]) for p in presets), + "app_mode": sum(p["app_mode"] for p in presets), "demo": bool(demo)} + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--out", type=Path, default=ROOT / "site") + parser.add_argument("--demo", type=Path, help="demo video, copied to static/media/demo.mp4") + args = parser.parse_args() + report = build(args.out, args.demo) + print(f"{args.out}: {report['presets']} effects, {report['apps']} with a Comfy Cloud app link " + f"({report['app_mode']} can run in App Mode), demo {'yes' if report['demo'] else 'no'}") + + +if __name__ == "__main__": + main() diff --git a/vfx4all/config.py b/vfx4all/config.py index 00bdaf3..c6f2b66 100644 --- a/vfx4all/config.py +++ b/vfx4all/config.py @@ -4,6 +4,7 @@ import os from pathlib import Path +from urllib.parse import urlparse ROOT = Path(__file__).resolve().parent.parent WORKFLOWS_DIR = ROOT / "workflows" @@ -47,6 +48,22 @@ def api_key() -> str: return os.environ.get("COMFY_API_KEY", "").strip() +def comfy_target() -> dict: + """Where the graphs run, from COMFY_BASE_URL (read by comfy-sdk): Comfy Cloud when unset, a + Comfy Developer Platform deployment, or your own ComfyUI behind comfy-api-proxy.""" + url = os.environ.get("COMFY_BASE_URL", "").strip() + host = urlparse(url if "//" in url else f"//{url}").hostname or "" + # label starts a sentence ("Your ComfyUI ready"), name goes inside one ("queued on your ComfyUI") + if not url or host == "comfy.org" or host.endswith(".comfy.org"): + return {"kind": "cloud", "label": "Comfy Cloud", "name": "Comfy Cloud", "needs_key": True} + if host.endswith(".run.comfy.app"): + return {"kind": "deployment", "label": "Your Comfy deployment", "name": "your Comfy deployment", "needs_key": True} + if host in ("localhost", "127.0.0.1", "::1") or host.endswith(".localhost"): + # comfy-api-proxy asks for no key unless it was started with --token + return {"kind": "local", "label": "Your ComfyUI", "name": "your ComfyUI", "needs_key": False} + return {"kind": "custom", "label": "Your Comfy server", "name": "your Comfy server", "needs_key": True} + + def _int_env(name: str, default: int) -> int: try: return int(os.environ.get(name, "").strip() or default) diff --git a/vfx4all/engine.py b/vfx4all/engine.py index 917b89f..41215ea 100644 --- a/vfx4all/engine.py +++ b/vfx4all/engine.py @@ -24,7 +24,7 @@ from comfy_sdk import Comfy, Progress, StatusChange from .config import (DFR_WORKFLOW, EDIT_WORKFLOW, EFFECT_WORKFLOW, FLF_WORKFLOW, SCENE_WORKFLOW, SIZE_MULTIPLE, - api_key, effect_pipeline) + api_key, comfy_target, effect_pipeline) from .jobs import ACTIVE_JOB, Cancelled ProgressFn = Callable[[float | None, str], None] @@ -139,9 +139,9 @@ def __init__(self, key: str | None = None) -> None: def client(self) -> Comfy: if self._client is None: key = self._key or api_key() - if not key: + if not key and comfy_target()["needs_key"]: raise MissingApiKey("Add your Comfy API key in Settings, or set COMFY_API_KEY in .env.") - self._client = Comfy(api_key=key) + self._client = Comfy(api_key=key or None) return self._client def close(self) -> None: @@ -223,17 +223,18 @@ def _run(self, wf, output_node: str, dest: Path, progress: ProgressFn) -> Path: owner = ACTIVE_JOB.get() if owner is not None and owner.cancel_requested: raise Cancelled() - progress(0.02, "Uploading inputs and queueing on Comfy Cloud") + target = comfy_target() + progress(0.02, f"Uploading inputs and queueing on {target['name']}") job = retry_io(lambda: self.client.submit(wf), retry_on=(httpx.ConnectError, httpx.ConnectTimeout)) if owner is not None: owner.attach(job) - progress(0.05, "Queued on Comfy Cloud") + progress(0.05, f"Queued on {target['name']}") try: for event in job.events(): if isinstance(event, Progress): progress(max(0.05, min(event.value, 0.98)), event.message or "Generating") elif isinstance(event, StatusChange): - progress(None, f"Comfy Cloud: {event.status}") + progress(None, f"{target['label']}: {event.status}") except Exception: # the live stream is best effort; polling below is authoritative pass try: diff --git a/vfx4all/jobs.py b/vfx4all/jobs.py index 2c2c0d7..695bbd4 100644 --- a/vfx4all/jobs.py +++ b/vfx4all/jobs.py @@ -11,6 +11,7 @@ import contextvars import json import logging +import re import shutil import threading import time @@ -25,7 +26,7 @@ from comfy_sdk.exceptions import (ComfyError, Forbidden, InsufficientCredits, InvalidWorkflow, JobFailed, MissingAsset, NotFound, QueueFull, Unauthorized) -from .config import RUNS_DIR, UPLOADS_DIR, daily_runs, hosted, ip_jobs, session_jobs +from .config import RUNS_DIR, UPLOADS_DIR, comfy_target, daily_runs, hosted, ip_jobs, session_jobs log = logging.getLogger("vfx4all") @@ -105,26 +106,58 @@ def output_paths(self) -> set[str]: return {url.removeprefix("/files/") for url in self.outputs.values()} +GUIDE = "docs/RUN_ON_YOUR_COMFYUI.md" +# How ComfyUI (directly or behind comfy-api-proxy) and Comfy deployments name a node or a model file +# the graph asks for and the server does not have. +MISSING_NODE = (re.compile(r"Cannot execute because (?:a )?node (?:type )?['\"]?([\w.\-]+?)['\"]? does not exist"), + re.compile(r"Node ['\"]([^'\"]+)['\"] not found", re.I), + re.compile(r"unknown node type:? ['\"]?([\w.\-]+)", re.I)) +MISSING_MODEL = re.compile(r"['\"]([^'\"]+\.(?:safetensors|ckpt|pt|pth|gguf|bin))['\"] not in") + + +def missing_part(exc: BaseException) -> str | None: + """A message naming the missing node or model, when the error says which one it is.""" + text = " ".join([str(getattr(exc, "message", "") or ""), json.dumps(getattr(exc, "details", None), default=str), str(exc)]) + for pattern in MISSING_NODE: + found = pattern.search(text) + if found: + return (f"This Comfy server has no {found.group(1)} node. VFX 4 All only uses core nodes, so update " + f"ComfyUI to its latest release and restart it ({GUIDE}).") + found = MISSING_MODEL.search(text) + if found: + return (f"This Comfy server does not have the model file {found.group(1)}. Download it as {GUIDE} " + f"explains, then restart ComfyUI.") + return None + + def friendly_error(exc: BaseException, job_id: str) -> str: """What the user reads when a job fails. The traceback goes to the log and error.txt.""" if isinstance(exc, Cancelled): return "Cancelled." + if isinstance(exc, ComfyError) and (missing := missing_part(exc)): + return missing + target = comfy_target() + where, Where = target["name"], target["label"] if isinstance(exc, Unauthorized): - return "Comfy Cloud rejected the API key. Check the key in Settings or in .env." + return f"{Where} rejected the API key. Check the key in Settings or in .env." if isinstance(exc, InsufficientCredits): return "The Comfy workspace of this API key is out of credits." if isinstance(exc, Forbidden): return "This Comfy API key is not allowed to run these models." if isinstance(exc, QueueFull): - return "Comfy Cloud is busy right now. Try again in a minute." + return f"{Where} is busy right now. Try again in a minute." if isinstance(exc, (InvalidWorkflow, MissingAsset, NotFound)): - return "Comfy Cloud could not run this graph; a model or node may have been renamed in its catalog." + if target["kind"] == "cloud": + return "Comfy Cloud could not run this graph; a model or node may have been renamed in its catalog." + return f"{Where} could not run this graph. Check that it has every node and model file ({GUIDE})." if isinstance(exc, JobFailed): - return "The render failed on Comfy Cloud. Try again or use another seed." + return f"The render failed on {where}. Try again or use another seed." if isinstance(exc, (httpx.TimeoutException, httpx.NetworkError)): - return "Comfy Cloud did not answer in time. Try again." + if target["kind"] == "local": + return f"Your ComfyUI did not answer. Check that ComfyUI and comfy-api-proxy are running ({GUIDE})." + return f"{Where} did not answer in time. Try again." if isinstance(exc, ComfyError): - return "Comfy Cloud returned an error. Try again in a minute." + return f"{Where} returned an error. Try again in a minute." # Our own errors already speak to the user: missing key, preset rules, every take black. if type(exc).__name__ in ("MissingApiKey", "MediaError") or isinstance(exc, (ValueError, KeyError, RuntimeError)): text = exc.args[0] if exc.args else str(exc) diff --git a/vfx4all/pipeline.py b/vfx4all/pipeline.py index 94d8c45..f1b5f25 100644 --- a/vfx4all/pipeline.py +++ b/vfx4all/pipeline.py @@ -16,7 +16,7 @@ closest_selector) from .engine import Engine from .jobs import Job -from .presets import get_preset, render, wardrobe_for +from .presets import get_preset, render, wardrobe_for, with_direction SPLICE_DURATION = 5 _SERVER_ENGINE: Engine | None = None @@ -119,7 +119,7 @@ def run_scene(job: Job, *, key: str | None, preset_id: str, refs: list[Path], su def run_effect(job: Job, *, key: str | None, preset_id: str, image: Path, source: str, - subject: str, aspect: str, quality: str, seed: int | None) -> None: + subject: str, aspect: str, quality: str, seed: int | None, direction: str = "") -> None: preset = get_preset(preset_id) if source == "scene": template, selector = preset["effect_prompt"], ASPECTS[aspect]["selector"] @@ -129,7 +129,7 @@ def run_effect(job: Job, *, key: str | None, preset_id: str, image: Path, source template = preset["generic_prompt"] info = media.probe(image) selector = closest_selector(info["width"], info["height"]) - prompt = render(template, subject=subject) + prompt = with_direction(render(template, subject=subject), direction) seed = _seed(seed) job.params.update(prompt=prompt, selector=selector) with engine_for(key) as engine: @@ -153,7 +153,7 @@ def run_effect(job: Job, *, key: str | None, preset_id: str, image: Path, source def run_splice(job: Job, *, key: str | None, preset_id: str, video: Path, seconds: float, - subject: str, quality: str, audio_mode: str, seed: int | None) -> None: + subject: str, quality: str, audio_mode: str, seed: int | None, direction: str = "") -> None: preset = get_preset(preset_id) if not supports(preset, "video"): raise ValueError(f"{preset['name']} cannot be spliced into a video yet") @@ -163,7 +163,7 @@ def run_splice(job: Job, *, key: str | None, preset_id: str, video: Path, second job.set(0.02, f"Grabbing the frame at {seconds:.2f} s") frame = media.extract_frame(video, seconds, job.dir / "cut_frame.png") job.add_output("frame", frame) - prompt = render(template, subject=subject) + prompt = with_direction(render(template, subject=subject), direction) selector = closest_selector(info["width"], info["height"]) job.params.update(prompt=prompt, selector=selector, cut=seconds) with engine_for(key) as engine: diff --git a/vfx4all/presets.py b/vfx4all/presets.py index c8e5749..1e24bc4 100644 --- a/vfx4all/presets.py +++ b/vfx4all/presets.py @@ -63,6 +63,19 @@ def render(template: str, *, subject: str, aspect_words: str = "16:9 landscape", return re.sub(r"\{(\w+)\}", lambda m: values.get(m.group(1), m.group(0)), template) +MAX_DIRECTION = 500 + + +def with_direction(prompt: str, direction: str) -> str: + """The user's extra direction goes after the preset's own effect prompt, as one more sentence.""" + extra = " ".join((direction or "").split())[:MAX_DIRECTION].strip() + if not extra: + return prompt + if extra[-1] not in ".!?": + extra += "." + return f"{prompt.rstrip()} {extra}" + + def wardrobe_for(preset: dict[str, Any], subject: str) -> str: wardrobe = preset.get("wardrobe") or {} return wardrobe.get(subject) or wardrobe.get("person") or "simple dark clothes" diff --git a/vfx4all/server.py b/vfx4all/server.py index 322c8e2..f173398 100644 --- a/vfx4all/server.py +++ b/vfx4all/server.py @@ -25,10 +25,10 @@ from pydantic import BaseModel, Field from . import __version__, media, pipeline -from .config import (ASPECTS, IMAGE_LIMITS, RUNS_DIR, UPLOADS_DIR, WEB_DIR, access_code, api_key, check_image, - check_video, hosted, load_dotenv, mode, retention_hours, video_limits) +from .config import (ASPECTS, IMAGE_LIMITS, ROOT, RUNS_DIR, UPLOADS_DIR, WEB_DIR, access_code, api_key, check_image, + check_video, comfy_target, hosted, load_dotenv, mode, retention_hours, video_limits) from .jobs import LimitReached, JobManager -from .presets import get_preset, load_presets, public_view +from .presets import MAX_DIRECTION, get_preset, load_presets, public_view load_dotenv() RUNS_DIR.mkdir(parents=True, exist_ok=True) @@ -149,7 +149,7 @@ def _upload_path(upload_id: str, request: Request) -> Path: def _key(user_key: str | None) -> str | None: """The key a generation bills: the user's own if they sent one, else the server's.""" key = (user_key or "").strip() or None - if not key and not api_key(): + if not key and not api_key() and comfy_target()["needs_key"]: raise HTTPException(401, "No Comfy API key. Add yours in Settings or set COMFY_API_KEY in .env.") return key @@ -178,7 +178,10 @@ def _submit(kind: str, params: dict, work, request: Request) -> dict: @app.get("/api/health") def health() -> dict: - return {"ok": True, "version": __version__, "mode": mode(), "has_key": bool(api_key()), + target = comfy_target() + return {"ok": True, "version": __version__, "mode": mode(), + "has_key": bool(api_key()) or not target["needs_key"], + "target": {"kind": target["kind"], "label": target["label"]}, "access_required": bool(access_code()), "video_limits": video_limits(), "image_limits": IMAGE_LIMITS} @@ -242,7 +245,8 @@ class EffectRequest(BaseModel): subject: Subject = "person" aspect: Aspect = "16:9" quality: Quality = "draft" - seed: int | None = None + seed: int | None = Field(None, ge=0, le=2**32 - 1) + direction: str = Field("", max_length=MAX_DIRECTION) # added to the end of the effect prompt class SpliceRequest(BaseModel): @@ -252,7 +256,8 @@ class SpliceRequest(BaseModel): subject: Subject = "person" quality: Quality = "draft" audio: Literal["effect", "original"] = "effect" - seed: int | None = None + seed: int | None = Field(None, ge=0, le=2**32 - 1) + direction: str = Field("", max_length=MAX_DIRECTION) @app.post("/api/jobs/scene") @@ -286,7 +291,7 @@ def effect(req: EffectRequest, request: Request, x_comfy_key: str | None = Heade raise HTTPException(400, "Send scene_job or image") return _submit("effect", req.model_dump(), lambda j: pipeline.run_effect( j, key=key, preset_id=req.preset, image=image, source=source, subject=req.subject, - aspect=req.aspect, quality=req.quality, seed=req.seed), request) + aspect=req.aspect, quality=req.quality, seed=req.seed, direction=req.direction), request) @app.post("/api/jobs/splice") @@ -298,7 +303,7 @@ def splice(req: SpliceRequest, request: Request, x_comfy_key: str | None = Heade video = _upload_path(req.video, request) return _submit("splice", req.model_dump(), lambda j: pipeline.run_splice( j, key=key, preset_id=req.preset, video=video, seconds=req.seconds, subject=req.subject, - quality=req.quality, audio_mode=req.audio, seed=req.seed), request) + quality=req.quality, audio_mode=req.audio, seed=req.seed, direction=req.direction), request) def _job_for(job_id: str, request: Request): @@ -337,6 +342,20 @@ def files(path: str, request: Request) -> FileResponse: return FileResponse(target) +SAMPLES_DIR = ROOT / "samples" +SAMPLE_TYPES = {".jpg", ".jpeg", ".png", ".webp", ".mp4"} + + +@app.get("/samples/{kind}/{name}") +def sample(kind: Literal["photos", "clips"], name: str) -> FileResponse: + """The photos and clips in samples/, for the "Try our sample" buttons.""" + folder = (SAMPLES_DIR / kind).resolve() + target = (folder / name).resolve() + if target.parent != folder or target.suffix.lower() not in SAMPLE_TYPES or not target.is_file(): + raise HTTPException(404, "Not found") + return FileResponse(target) + + app.mount("/static", StaticFiles(directory=WEB_DIR), name="static") diff --git a/web/app.css b/web/app.css index 64566d6..6480d42 100644 --- a/web/app.css +++ b/web/app.css @@ -181,7 +181,13 @@ h1 { margin: 0 0 18px; font-size: clamp(38px, 6.4vw, 78px); font-weight: 900; li /* ------------------------------------------------ sections */ section { position: relative; } -.effects, .studio, .how { padding: 70px clamp(16px, 5vw, 64px) 30px; max-width: 1480px; margin: 0 auto; } +.effects, .studio, .how, .try, .demo { padding: 70px clamp(16px, 5vw, 64px) 30px; max-width: 1480px; margin: 0 auto; } + +/* Static catalog site (tools/build_site.py): no server, so the studio and the key controls go away */ +html:has(meta[name="vfx4all-mode"][content="static"]) .app-only { display: none !important; } +.try-preview, .demo-video { display: block; width: 100%; aspect-ratio: 16 / 9; background: #000; object-fit: cover; } +.demo-video { object-fit: contain; } +.try-note { margin: 0; font-size: 13px; line-height: 1.5; } .sec-head { display: flex; align-items: end; justify-content: space-between; gap: 20px; margin-bottom: 22px; } .eyebrow { margin: 0 0 6px; font-size: 13px; font-weight: 800; letter-spacing: 0.2em; text-transform: uppercase; color: var(--text-2); display: flex; align-items: center; gap: 12px; } .eyebrow::after { content: ""; width: 64px; height: 2px; background: linear-gradient(to right, var(--magenta), transparent); } @@ -298,7 +304,16 @@ section { position: relative; } .field { display: flex; flex-direction: column; gap: 7px; } .label { font-size: 11px; font-weight: 800; letter-spacing: 0.12em; text-transform: uppercase; color: var(--muted); } .label em { font-style: normal; font-weight: 600; letter-spacing: 0; text-transform: none; opacity: 0.8; } -textarea, input[type="password"] { width: 100%; padding: 10px 12px; background: rgba(0, 0, 0, 0.3); border: 1px solid var(--line); resize: vertical; } +textarea, input[type="password"], input[type="number"] { width: 100%; padding: 10px 12px; background: rgba(0, 0, 0, 0.3); border: 1px solid var(--line); resize: vertical; } +.sample-btn { align-self: flex-start; margin-top: -6px; padding: 0; border: 0; background: none; cursor: pointer; + color: var(--cyan); font-size: 12.5px; font-weight: 700; text-decoration: underline; text-underline-offset: 3px; } +.sample-btn:hover { color: var(--text); } +.sample-btn:disabled { opacity: 0.5; cursor: wait; } +.more { padding: 10px 12px; border: 1px solid var(--line); } +.more summary { cursor: pointer; width: fit-content; font-size: 11px; font-weight: 800; letter-spacing: 0.12em; text-transform: uppercase; color: var(--muted); } +.more summary:hover { color: var(--text); } +.more[open] summary { margin-bottom: 12px; } +.more .field + .field { margin-top: 12px; } textarea:focus, input:focus { outline: none; border-color: var(--magenta); box-shadow: 0 0 0 3px rgba(255, 0, 255, 0.15); } .row { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; } .seg { display: flex; gap: 6px; flex-wrap: wrap; } diff --git a/web/app.js b/web/app.js index e6399ed..9f41e24 100644 --- a/web/app.js +++ b/web/app.js @@ -4,6 +4,8 @@ const $ = (sel, el = document) => el.querySelector(sel); const $$ = (sel, el = document) => [...el.querySelectorAll(sel)]; const TOUCH = matchMedia("(hover: none)").matches; const REDUCED = matchMedia("(prefers-reduced-motion: reduce)").matches; +// "static" on the catalog site built by tools/build_site.py: no server behind it, effects run as Comfy Cloud apps. +const STATIC = document.querySelector('meta[name="vfx4all-mode"]')?.content === "static"; const CATEGORY_COLOR = { Elements: "#ff00ff", Transformations: "#33f5ff", Clones: "#cc66ff", Worlds: "#38b6ff", @@ -211,6 +213,7 @@ function hexGrid(canvas) { /* ---------- effects rail ---------- */ function presetAllowed(p) { + if (STATIC) return true; if (state.tab === "video") return p.modes.includes("video"); if (state.ownImage) return p.modes.includes("own_image"); return p.modes.includes("scene"); @@ -298,7 +301,7 @@ function card(item) { if (!allowed) return toast(whyNot(item)); const again = state.preset === item.id; selectPreset(item.id); - if (again || e.target.closest(".card-use")) $("#studio").scrollIntoView({ behavior: "smooth" }); + if (again || e.target.closest(".card-use")) $(STATIC ? "#try" : "#studio").scrollIntoView({ behavior: "smooth" }); }, onmouseenter: () => { if (!TOUCH) video.play().then(() => node.classList.add("playing")).catch(() => {}); }, onmouseleave: () => { if (!TOUCH) { video.pause(); node.classList.remove("playing"); } }, @@ -315,12 +318,17 @@ function card(item) { el("span", { class: "card-name" }, [icon(item.icon || "sparkles"), document.createTextNode(item.name)]), el("span", { class: "card-desc", text: item.description }), el("span", { class: "card-use" }, [icon(state.preset === item.id ? "check" : "wand", "ic sm"), - document.createTextNode(state.preset === item.id ? "Selected, go to studio" : "Use this effect")]), + document.createTextNode(useLabel(state.preset === item.id))]), ]), ])); return node; } +function useLabel(on) { + if (STATIC) return on ? "Selected, see how to run it" : "Try this effect"; + return on ? "Selected, go to studio" : "Use this effect"; +} + function selectPreset(id) { state.preset = id; const p = state.presets.find((x) => x.id === id); @@ -329,8 +337,9 @@ function selectPreset(id) { c.classList.toggle("selected", on); c.setAttribute("aria-selected", String(on)); const use = c.querySelector(".card-use"); - if (use) use.replaceChildren(icon(on ? "check" : "wand", "ic sm"), document.createTextNode(on ? "Selected, go to studio" : "Use this effect")); + if (use) use.replaceChildren(icon(on ? "check" : "wand", "ic sm"), document.createTextNode(useLabel(on))); }); + if (STATIC) paintTry(p); $("#chosenName").textContent = p ? p.name : "Pick one above"; const thumb = $("#chosenThumb"); if (p) thumb.src = p.thumb; else thumb.removeAttribute("src"); @@ -339,6 +348,26 @@ function selectPreset(id) { if (p?.comfy_app) link.href = p.comfy_app; } +// Static site: the chosen effect, its preview, and its Comfy Cloud app when it has one. +function paintTry(p) { + $("#tryName").textContent = p ? p.name : "Pick one above"; + const thumb = $("#tryThumb"); + if (p) thumb.src = p.thumb; else thumb.removeAttribute("src"); + const preview = $("#tryPreview"); + if (p && preview.getAttribute("src") !== p.preview) { + preview.src = p.preview; + if (!REDUCED) preview.play().catch(() => {}); + } + const run = $("#tryRun"), note = $("#tryNote"); + run.classList.toggle("hidden", !p?.comfy_app); + if (p?.comfy_app) run.href = p.comfy_app; + const why = !p || p.comfy_app ? "" : p.app_mode + ? "Its Comfy Cloud app is on its way." + : `${p.name} needs more than one graph, so it runs only in the full app.`; + note.textContent = why; + note.classList.toggle("hidden", !why); +} + function updateRailBar() { const rail = $("#rail"); const total = rail.scrollWidth || 1; @@ -446,10 +475,20 @@ function promptBox(text) { // The request fields a result can be run again with, by job kind. const RERUN_FIELDS = { - effect: ["preset", "scene_job", "image", "subject", "aspect", "quality"], - splice: ["preset", "video", "seconds", "subject", "quality", "audio"], + effect: ["preset", "scene_job", "image", "subject", "aspect", "quality", "direction"], + splice: ["preset", "video", "seconds", "subject", "quality", "audio", "direction"], }; +// "More options": the extra direction and a seed, sent only when filled in. +function moreOptions() { + const out = {}; + const direction = $("#direction").value.trim(); + if (direction) out.direction = direction; + const seed = $("#seedInput").value.trim(); + if (/^\d+$/.test(seed) && Number(seed) <= 4294967295) out.seed = Number(seed); + return out; +} + function resultTitle(kind, body) { const where = kind === "splice" ? ` from ${Number(body.seconds || 0).toFixed(2)} s` : ""; return `${presetName(body.preset)}${where} | ${qualityLabel(body.quality)}`; @@ -623,6 +662,26 @@ async function addPhotos(fileList) { try { state.refs.push(...(await uploadFiles(images.slice(0, 2 - state.refs.length)))); renderThumbs(); } catch (err) { fail(err); } } +// The files in samples/: photos and a clip of our own team, who agreed to be used. +async function sampleFiles(paths) { + return Promise.all(paths.map(async (path) => { + const res = await fetch(path); + if (!res.ok) throw new Error("This copy of the app has no samples folder."); + const blob = await res.blob(); + return new File([blob], path.split("/").pop(), { type: blob.type }); + })); +} + +async function useSample(button, paths, add) { + button.disabled = true; + try { + const files = await sampleFiles(paths); + $("#consent").checked = true; + await add(files); + toast("Sample of our own team, used with their permission"); + } catch (err) { fail(err); } finally { button.disabled = false; } +} + function needPhoto() { if (!state.preset) { toast("Pick an effect first", true); $("#effects").scrollIntoView({ behavior: "smooth" }); return false; } if (!state.refs.length) { toast("Add a photo first", true); return false; } @@ -640,16 +699,17 @@ function showScene(job) { } async function buildScene() { + const { seed } = moreOptions(); // undefined drops out of the JSON body const done = await waitJob(await startJob("scene", { preset: state.preset, refs: state.refs.map((r) => r.id), subject: state.photo.subject, - notes: $("#notes").value, aspect: state.photo.aspect, + notes: $("#notes").value, aspect: state.photo.aspect, seed, }), { title: presetName(), kind: "scene" }); showScene(done); return done; } function animate() { - const body = { preset: state.preset, subject: state.photo.subject, aspect: state.photo.aspect, quality: state.photo.quality }; + const body = { preset: state.preset, subject: state.photo.subject, aspect: state.photo.aspect, quality: state.photo.quality, ...moreOptions() }; if (state.ownImage) body.image = state.refs[0].id; else body.scene_job = state.sceneJob; return runJob("effect", body, resultTitle("effect", body)); } @@ -731,7 +791,7 @@ function splice() { if (!state.upload) return toast("Add a clip first", true); const body = { preset: state.preset, video: state.upload.id, seconds: state.cut, - subject: state.video.subject, quality: state.video.quality, audio: state.video.audio, + subject: state.video.subject, quality: state.video.quality, audio: state.video.audio, ...moreOptions(), }; return runJob("splice", body, resultTitle("splice", body)); } @@ -806,7 +866,9 @@ async function refreshKeyStatus() { const own = Boolean(userKey()); const needsCode = health.access_required && !accessCode(); pill.className = `status ${!needsCode && (own || health.has_key) ? "ok" : "warn"}`; - text = needsCode ? "Access code needed" : own ? "Your Comfy key" : health.has_key ? "Comfy Cloud ready" : "No Comfy key"; + // where the graphs run: Comfy Cloud, your own ComfyUI or a Comfy deployment (COMFY_BASE_URL) + const where = health.target?.label || "Comfy Cloud"; + text = needsCode ? "Access code needed" : own ? "Your Comfy key" : health.has_key ? `${where} ready` : "No Comfy key"; } catch { pill.className = "status warn"; text = "Server offline"; @@ -830,6 +892,9 @@ async function init() { bindRail(); bindDrop($("#photoDrop"), $("#photoInput"), addPhotos); bindDrop($("#videoDrop"), $("#videoInput"), addVideo); + $("#samplePhotos").addEventListener("click", (e) => useSample(e.currentTarget, + ["/samples/photos/front.jpg", "/samples/photos/side.jpg"], (files) => { state.refs = []; renderThumbs(); return addPhotos(files); })); + $("#sampleClip").addEventListener("click", (e) => useSample(e.currentTarget, ["/samples/clips/street-walk-16x9.mp4"], addVideo)); $$(".mode").forEach((m) => m.addEventListener("click", () => switchTab(m.dataset.tab))); $("#ownImage").addEventListener("change", (e) => { @@ -866,8 +931,14 @@ async function init() { }); paintLimits(); + if (STATIC) { + // the demo section only shows when the site was built with a demo video in it + $$(".static-only").forEach((n) => { if (n.id !== "demo" || $("video", n)) n.hidden = false; }); + $("#heroCta").href = "#try"; + } try { - const [presets, upcoming] = await Promise.all([api("/api/presets"), fetch("/static/upcoming.json").then((r) => r.json()).catch(() => [])]); + const presetList = STATIC ? fetch("/static/presets.json").then((r) => r.json()) : api("/api/presets"); + const [presets, upcoming] = await Promise.all([presetList, fetch("/static/upcoming.json").then((r) => r.json()).catch(() => [])]); state.presets = presets; state.upcoming = upcoming; const total = presets.length + upcoming.length; @@ -879,6 +950,7 @@ async function init() { renderRail(); if (presets[0]) selectPreset(presets[0].id); } catch (err) { toast(`Could not load effects: ${err.message}`, true); } + if (STATIC) return; await refreshKeyStatus(); restore(); } diff --git a/web/index.html b/web/index.html index e587992..dd31bc3 100644 --- a/web/index.html +++ b/web/index.html @@ -6,6 +6,8 @@ VFX 4 All + + @@ -75,10 +77,12 @@ -
+
Checking
@@ -99,7 +103,7 @@

Cinematic VFX
for everyone

Drop a photo or a clip, pick an effect, and get a film grade shot with sound. Every effect builds its own scene around the people in it.

- Start creating + Start creating Browse 45 effects
    @@ -129,8 +133,42 @@

    Effects

+ + + + + + -
+

Studio

@@ -171,6 +209,7 @@

Studio

A clear front view works best. A second angle helps the likeness. PNG, JPG or WEBP. +
+
    @@ -274,6 +314,18 @@

    Studio

+
+ More options + + +
+