| 1 | # Local R2V scheduler |
| 2 | |
| 3 | The root `server.py` is the command-line entry point for a local inference |
| 4 | service, not a proxy. Its implementation is organized under `server/`, while |
| 5 | model execution stays in this repository through `InferenceEngine`. |
| 6 | |
| 7 | ## Runtime model |
| 8 | |
| 9 | R2V and merge jobs are scheduled from process memory. Every state transition is |
| 10 | mirrored to a SQLite recovery journal, which is not polled as a work queue. |
| 11 | There is one worker thread and one `LocalModelRuntime` per configured logical |
| 12 | CUDA device; model objects are never shared between GPUs. Each free worker |
| 13 | atomically claims one FIFO job, so a worker cannot reserve a batch and leave |
| 14 | other GPUs idle. |
| 15 | |
| 16 | ```text |
| 17 | POST /r2v |
| 18 | -> in-memory FIFO |
| 19 | -> live GPU/RAM admission |
| 20 | -> one job per free GPU -> conditioning-cache lookup |
| 21 | -> cache miss: retain generator on GPU when VRAM permits |
| 22 | -> otherwise retain on CPU when RAM permits, otherwise release |
| 23 | -> encode text/image/audio -> save cache |
| 24 | -> cache hit: reuse current weight tier |
| 25 | -> load selected release checkpoint once when needed |
| 26 | -> sequential DMD inference -> tiled VAE decode -> MP4 write |
| 27 | -> artifact index -> optional callback |
| 28 | ``` |
| 29 | |
| 30 | Server deployment and model inference use separate YAML files. Start the |
| 31 | service with a server YAML; its `inference.config` field selects the inference |
| 32 | YAML, and that inference YAML selects the checkpoint under `paths.checkpoint`. |
| 33 | A server process serves one checkpoint configuration; run separate processes |
| 34 | with separate ports and GPU sets when several checkpoint variants must be |
| 35 | available simultaneously. |
| 36 | |
| 37 | Environment variables such as `ECHO_INFERENCE_CONFIG` and `ECHO_CHECKPOINT` |
| 38 | remain supported as explicit deployment overrides, but are not needed for the |
| 39 | normal YAML-driven startup path. |
| 40 | |
| 41 | The server defaults to `auto` DiT residency. It compares live free VRAM |
| 42 | with the measured whole-device peak for the selected release checkpoint plus a |
| 43 | device-relative reserve. A sufficiently large GPU uses the normal fully |
| 44 | resident DiT and keeps it on GPU through tiled decode and subsequent cache-hit |
| 45 | requests. A smaller or occupied GPU enables layerwise swap and transfers only |
| 46 | the active/prefetched transformer blocks to CUDA. `resident` and `swap` force a |
| 47 | profile for explicit performance testing. The server YAML's |
| 48 | `runtime.dit_residency` overrides the inference YAML's offload flag; the |
| 49 | standalone `inference.py` CLI continues to obey its inference YAML exactly. It |
| 50 | never changes checkpoint precision. |
| 51 | |
| 52 | `configs/server.consumer.yaml` selects the standalone FP4 inference profile and |
| 53 | forces layerwise swap. This is the recommended consumer-GPU deployment: on the |
| 54 | measured RTX 5090 workload it used 16.29 GiB peak VRAM and 26.51 GiB peak |
| 55 | process RAM. Tiled VAE decode remains enabled (`64` frames / `512` pixels with |
| 56 | the configured overlaps). |
| 57 | |
| 58 | ## Scheduling and weight lifecycle |
| 59 | |
| 60 | A worker probes CUDA with `torch.cuda.mem_get_info()` and host memory with |
| 61 | `psutil.virtual_memory()` before claiming work. Headroom is derived from the |
| 62 | reported device/system totals (5% GPU and 10% RAM by default), rather than fixed |
| 63 | GiB thresholds. The ratios can be overridden with |
| 64 | `ECHO_GPU_HEADROOM_FRACTION` and `ECHO_RAM_HEADROOM_FRACTION`. |
| 65 | Cold-load admission additionally accounts for the selected checkpoint's actual |
| 66 | file size and precision format; a warm runtime uses its measured in-memory tensor |
| 67 | storage and current GPU/CPU location. The calculated requirements and mode are |
| 68 | reported under each worker's `/health` resource snapshot. |
| 69 | The resolved `resident` or `swap` choice and its observed/required GiB values |
| 70 | are included in `admission_plan.dit_residency`. |
| 71 | |
| 72 | The generator and decode VAEs remain on GPU across cache hits whenever capacity |
| 73 | allows, including across the decode-to-next-denoise boundary. Cache-miss |
| 74 | conditioning is serialized across GPUs so several Gemma copies cannot consume |
| 75 | host RAM simultaneously. If live free VRAM can hold the conditioning stack, |
| 76 | generation weights stay on GPU; otherwise they move to CPU when RAM permits, or |
| 77 | are released as the last fallback. Decode applies the same GPU -> CPU -> release |
| 78 | tiering. OOM always releases all weights. The default idle timeout is `0`, which |
| 79 | disables idle eviction; set `ECHO_MODEL_IDLE_SECONDS` only when sharing GPUs. |
| 80 | MSST is bound to the worker's logical CUDA device and is released before a cold |
| 81 | generator load. |
| 82 | |
| 83 | ## Task state |
| 84 | |
| 85 | The externally visible job status is `queued`, `running`, `succeeded`, or |
| 86 | `failed`. `stage` gives the finer transition: |
| 87 | |
| 88 | ```text |
| 89 | queued -> claimed -> validating -> conditioning_cache_lookup |
| 90 | -> [keeping_generator_on_gpu | staging_generator_on_cpu |
| 91 | | unloading_for_conditioning -> conditioning] |
| 92 | -> loading_generator -> inferring -> decoding -> writing |
| 93 | -> succeeded | failed |
| 94 | ``` |
| 95 | |
| 96 | Queued jobs survive process restart. A job interrupted while running is restored |
| 97 | to the FIFO with stage `recovered`; completed/failed status and pending callback |
| 98 | retries are restored without rerunning the job. Agent `job_id` is a durable |
| 99 | idempotency key: retrying the same request returns the original `version_id`, |
| 100 | while reusing it for a different payload returns HTTP 409. `GET /health` exposes |
| 101 | each GPU worker's task, weight location, latest GPU/RAM snapshot, and the |
| 102 | `scheduler` counters consumed by Echo Director admission. `GET /version/{id}` |
| 103 | continues to work after restart. |
| 104 | |
| 105 | ## Input boundary |
| 106 | |
| 107 | The local endpoint accepts up to seven ordered memory slots. Resources may be |
| 108 | base64 `data:` URLs, absolute HTTP(S) URLs, `file://` URLs, or absolute local |
| 109 | paths visible to the server. Inline data is written to a content-addressed local |
| 110 | asset before queueing. A bare memory `shot_id` resolves to the newest successful |
| 111 | local R2V artifact with the same `work_id`; the server extracts a representative |
| 112 | frame and its complete audio track, then submits those assets through the same |
| 113 | conditioning path as explicit `image_url` and `audio_url` inputs. An unavailable |
| 114 | shot reference returns HTTP 409. |
| 115 | |
| 116 | ## HTTP API |
| 117 | |
| 118 | The examples below assume the default base URL, `http://127.0.0.1:8221`. |
| 119 | Interactive OpenAPI documentation is available at `/docs` while the server is |
| 120 | running. |
| 121 | |
| 122 | ### `GET /health` |
| 123 | |
| 124 | Returns service readiness, R2V and merge queue counts, GPU-worker occupancy, |
| 125 | the active inference configuration, memory-residency policy, and FFmpeg |
| 126 | availability. |
| 127 | |
| 128 | ```bash |
| 129 | curl http://127.0.0.1:8221/health |
| 130 | ``` |
| 131 | |
| 132 | Important response fields: |
| 133 | |
| 134 | - `status`: `ok` after both services have started. |
| 135 | - `inference.enabled`: whether GPU inference workers are enabled. |
| 136 | - `inference.workers`: per-GPU state, current task, model location, and latest |
| 137 | resource snapshot. |
| 138 | - `queues.r2v` and `queues.merge`: counts grouped by `queued`, `running`, |
| 139 | `succeeded`, and `failed`. |
| 140 | - `scheduler`: aggregate worker and pending-task counters used by Echo Director. |
| 141 | - `ffmpeg_available`: whether the configured FFmpeg executable can be found. |
| 142 | |
| 143 | ### `POST /r2v` |
| 144 | |
| 145 | Queues one reference-to-video generation job. |
| 146 | |
| 147 | ```bash |
| 148 | curl -X POST http://127.0.0.1:8221/r2v \ |
| 149 | -H 'Content-Type: application/json' \ |
| 150 | -d '{ |
| 151 | "work_id": "demo", |
| 152 | "shot_id": "shot-002", |
| 153 | "job_id": "demo-shot-002-v1", |
| 154 | "prompt": "A cinematic tracking shot through a rainy city street.", |
| 155 | "condition_img": "/absolute/path/to/first-frame.png", |
| 156 | "memory_slots": [ |
| 157 | {"shot_id": "shot-001"} |
| 158 | ], |
| 159 | "num_frames": 241, |
| 160 | "width": 1280, |
| 161 | "height": 736, |
| 162 | "seed": 42 |
| 163 | }' |
| 164 | ``` |
| 165 | |
| 166 | Request fields: |
| 167 | |
| 168 | | Field | Required | Description | |
| 169 | | --- | --- | --- | |
| 170 | | `work_id` | yes | Logical project or sequence identifier. | |
| 171 | | `shot_id` | yes | Identifier for the shot being generated. | |
| 172 | | `prompt` | yes | Non-empty generation prompt. | |
| 173 | | `memory_slots` | yes | Ordered list of up to seven memory references; it may be empty. | |
| 174 | | `job_id` | no | Durable idempotency key supplied by the caller. | |
| 175 | | `condition_img` | no | First-frame image as an HTTP(S) URL, `file://` URL, absolute path, or base64 data URL. | |
| 176 | | `callback_url` | no | HTTP(S) endpoint notified when the job succeeds or fails. | |
| 177 | | `callback_context` | no | Object copied into supported callback context fields. | |
| 178 | | `num_frames` | no | Positive frame count; otherwise the inference YAML default is used. | |
| 179 | | `duration_sec` | no | Positive target duration metadata. | |
| 180 | | `width`, `height` | no | Positive output dimensions; otherwise configuration defaults are used. | |
| 181 | | `seed` | no | Integer generation seed. | |
| 182 | |
| 183 | Each `memory_slots` entry must contain exactly one of: |
| 184 | |
| 185 | - `shot_id`: resolve the newest successful local R2V artifact with the same |
| 186 | `work_id`; or |
| 187 | - `image_url`: use an explicit image resource, optionally paired with |
| 188 | `audio_url` or `audio_mode: "empty"`. |
| 189 | |
| 190 | `image_mode` is currently unsupported. A slot using `shot_id` cannot also |
| 191 | provide audio fields. `X-Nanobot-Director-Callback-Url` may be supplied as a |
| 192 | request header and takes precedence over `callback_url`. |
| 193 | |
| 194 | The endpoint returns the accepted task immediately. `task_id`, `version_id`, |
| 195 | and `remote_task_id` identify the same R2V job. Poll `status_url` until `status` |
| 196 | is `succeeded` or `failed`. |
| 197 | |
| 198 | ```json |
| 199 | { |
| 200 | "accepted": true, |
| 201 | "kind": "r2v", |
| 202 | "task_id": "8d1d...", |
| 203 | "version_id": "8d1d...", |
| 204 | "remote_task_id": "8d1d...", |
| 205 | "work_id": "demo", |
| 206 | "job_id": "demo-shot-002-v1", |
| 207 | "shot_id": "shot-002", |
| 208 | "status": "queued", |
| 209 | "stage": "queued", |
| 210 | "queue_position": 1, |
| 211 | "status_url": "http://127.0.0.1:8221/version/8d1d...", |
| 212 | "artifacts": [] |
| 213 | } |
| 214 | ``` |
| 215 | |
| 216 | Reusing a `job_id` with the same request returns the original `version_id`. |
| 217 | Reusing it with a different request returns HTTP 409. |
| 218 | |
| 219 | ### `POST /merge` |
| 220 | |
| 221 | Queues an ordered MP4 concatenation job. Each shot must provide either a local |
| 222 | R2V `version_id` or an absolute HTTP(S) `video_url`. When both are present, |
| 223 | `version_id` is used. |
| 224 | |
| 225 | ```bash |
| 226 | curl -X POST http://127.0.0.1:8221/merge \ |
| 227 | -H 'Content-Type: application/json' \ |
| 228 | -d '{ |
| 229 | "work_id": "demo", |
| 230 | "job_id": "demo-merge-v1", |
| 231 | "shots": [ |
| 232 | {"version_id": "8d1d..."}, |
| 233 | {"video_url": "https://example.com/shot-003.mp4"} |
| 234 | ] |
| 235 | }' |
| 236 | ``` |
| 237 | |
| 238 | The response uses the same asynchronous pattern as `/r2v`, with `kind` set to |
| 239 | `merge`. On success, `video_url` and the `artifacts` list identify the merged |
| 240 | output. The endpoint also accepts the Echo Director envelope fields `payload`, |
| 241 | `job.job_id`, and `callback`; a callback URL may instead be supplied through |
| 242 | `X-Nanobot-Director-Callback-Url`. |
| 243 | |
| 244 | ### `GET /version/{version_id}` |
| 245 | |
| 246 | Returns the current R2V or merge job record. R2V responses include `stage`, |
| 247 | `queue_position`, worker/resource information, generated video fields, and |
| 248 | artifact metadata. Merge responses include the merged `video_url`. Unknown IDs |
| 249 | return HTTP 404. |
| 250 | |
| 251 | ```bash |
| 252 | curl http://127.0.0.1:8221/version/8d1d... |
| 253 | ``` |
| 254 | |
| 255 | ### `GET /artifact/{artifact_id}` |
| 256 | |
| 257 | Returns metadata for one generated artifact, including its `version_id`, |
| 258 | `work_id`, optional `shot_id`, `kind`, `role`, public URL, byte size, SHA-256, |
| 259 | timestamps, and generation metadata. Local filesystem paths are not exposed. |
| 260 | Unknown IDs return HTTP 404. |
| 261 | |
| 262 | ```bash |
| 263 | curl http://127.0.0.1:8221/artifact/2b57... |
| 264 | ``` |
| 265 | |
| 266 | ### `GET /media/{asset_path}` |
| 267 | |
| 268 | Streams an MP4 stored below the configured `media_root`. Paths are constrained |
| 269 | to that directory. Missing files and traversal attempts return HTTP 404. |
| 270 | |
| 271 | ```bash |
| 272 | curl -O http://127.0.0.1:8221/media/r2v/demo/shot-002/8d1d.../output.mp4 |
| 273 | ``` |
| 274 | |
| 275 | ### Callbacks and errors |
| 276 | |
| 277 | When a callback URL is present, the server sends one JSON `POST` after success |
| 278 | or failure. R2V callbacks contain `work_id`, `job_id`, `shot_id`, |
| 279 | `remote_task_id`, `status`, and either result URLs or `error`. Merge callbacks |
| 280 | contain the corresponding job identifiers and a `result` object with the |
| 281 | merged URL. Failed callbacks are retried according to |
| 282 | `server.callback_max_attempts`; `0` means unlimited retries. |
| 283 | |
| 284 | Common HTTP errors are: |
| 285 | |
| 286 | | Status | Meaning | |
| 287 | | --- | --- | |
| 288 | | `404` | Task, artifact, or media file was not found. | |
| 289 | | `409` | Missing referenced shot or conflicting reuse of an idempotency key. | |
| 290 | | `422` | Invalid request fields, resource URL/path, or unsupported media input. | |
| 291 | | `503` | Service is not ready or the selected queue is full. | |
| 292 | |
| 293 | ## Start the server |
| 294 | |
| 295 | Run the checked-in server configuration directly: |
| 296 | |
| 297 | ```bash |
| 298 | uv run python server.py --config configs/server.consumer.yaml |
| 299 | ``` |
| 300 | |
| 301 | The server YAML owns the bind address, port and scheduler/runtime settings. It |
| 302 | must use one Uvicorn process because GPU workers live in process memory. GPU IDs |
| 303 | are logical IDs after `CUDA_VISIBLE_DEVICES` is applied; listing several IDs |
| 304 | explicitly opts those devices into the service. |
| 305 |