| 1 | # R2V request and conditioning boundary |
| 2 | |
| 3 | The public local entrypoint uses the same request concepts as the online R2V |
| 4 | service. One JSON file describes one output shot. `memory_slots` is an ordered |
| 5 | array of zero to seven references; `condition_img` is an independent clean first |
| 6 | frame and never consumes a memory slot. |
| 7 | |
| 8 | The canonical machine-readable contract is |
| 9 | [`schemas/r2v_request.schema.json`](../schemas/r2v_request.schema.json). Local |
| 10 | paths are resolved relative to the JSON file. HTTP(S) resources are downloaded |
| 11 | during condition encoding. For reproducible offline CLI inference, each memory |
| 12 | slot must use `image_url`. The local server additionally accepts a bare `shot_id` |
| 13 | and resolves it from its persisted successful artifacts for the same `work_id`. |
| 14 | It extracts a representative frame and the complete audio track before entering |
| 15 | this common conditioning boundary. |
| 16 | |
| 17 | ## Stage boundary |
| 18 | |
| 19 | `python inference.py --condition-encode` deliberately stops immediately before |
| 20 | the DiT. It performs these operations in order: |
| 21 | |
| 22 | 1. Load and validate every request, preserve memory-slot order, and truncate the |
| 23 | prompt to the online 1,500-character limit. |
| 24 | 2. Deduplicate prompts/assets and batch compatible work. Text is processed by |
| 25 | language-only Gemma and the Echo video/audio connectors. Images are resized |
| 26 | to the request's exact output size and encoded by the video VAE. Memory audio |
| 27 | is voice-filtered by MSST, normalized to stereo, and encoded by the audio VAE. |
| 28 | 3. Assemble per-request CPU tensors. Empty audio slots receive zero latents only |
| 29 | when another slot has real audio, so audio and video slot positions remain |
| 30 | aligned. |
| 31 | 4. Atomically write one `.safetensors` file per request. |
| 32 | |
| 33 | The cache contains: |
| 34 | |
| 35 | - `text.video_context`, optional `text.audio_context`, and `text.attention_mask`; |
| 36 | - optional `first_frame_latent` with shape `[1, 1, C, H, W]`; |
| 37 | - optional ordered `memory_video` with shape `[1, slots, C, H, W]`; |
| 38 | - optional `memory_audio` and `memory_audio_timestep`; |
| 39 | - audio segment lengths and slot-center RoPE parameters in metadata; |
| 40 | - request, checkpoint, Gemma, and input-content fingerprints. |
| 41 | |
| 42 | Generation loads this bundle, moves only the needed tensors to the DiT device, |
| 43 | and applies the online DMD behavior. If a first-frame latent exists, frame zero |
| 44 | uses timestep zero and is restored after every prediction and re-noise step. |
| 45 | Memory RoPE uses `slot_center`, offset `500`, and stride `50`. |
| 46 | |
| 47 | ## Commands |
| 48 | |
| 49 | Encode all Last Visa conditions, optionally sharded over GPUs: |
| 50 | |
| 51 | ```bash |
| 52 | torchrun --standalone --nproc-per-node=3 inference.py \ |
| 53 | --config configs/inference.fp8.yaml \ |
| 54 | --condition-encode \ |
| 55 | --conditioning-cache-dir conditioning_cache/the_last_visa |
| 56 | ``` |
| 57 | |
| 58 | Run one request from the complete cache: |
| 59 | |
| 60 | ```bash |
| 61 | python inference.py \ |
| 62 | --config configs/inference.fp8.yaml \ |
| 63 | --request examples/the_last_visa/requests/009_01_shot_008_nathan_replies_to_elena_r2v.json \ |
| 64 | --conditioning-cache-dir conditioning_cache/the_last_visa |
| 65 | ``` |
| 66 | |
| 67 | The old `--text-encode` spelling remains an alias, but it now executes the whole |
| 68 | conditioning stage rather than producing a text-only artifact. |
| 69 |