| 1 | # BF16, FP8 and FP4 inference |
| 2 | |
| 3 | Echo 1.5 uses one inference pipeline with three public generator precisions. |
| 4 | Only model construction is precision-specific. |
| 5 | |
| 6 | | Release checkpoint | Precision | Loader backend | |
| 7 | | --- | --- | --- | |
| 8 | | `echo15_full_dmd` | BF16 | complete DMD-merged model | |
| 9 | | `echo15_fp8` | FP8 | E4M3 weights, checkpoint scales, dynamic activation scaling and `torch._scaled_mm` | |
| 10 | | `echo15_fp4` | FP4 | ModelOpt LTX-2 plugin with packed FP4 transformer weights | |
| 11 | |
| 12 | All three are self-contained directory checkpoints selected by one |
| 13 | `paths.checkpoint` value. The manifest binds the public name, precision and |
| 14 | internal files together. |
| 15 | |
| 16 | ## BF16 |
| 17 | |
| 18 | Use `configs/inference.bf16.yaml`. DMD is merged once while preparing the release; |
| 19 | runtime directly loads `echo15_full_dmd/model.safetensors`. |
| 20 | |
| 21 | ## FP8 |
| 22 | |
| 23 | Use `configs/inference.fp8.yaml`. Before model |
| 24 | construction, the loader reads the safetensors header and discovers every |
| 25 | Linear layer that has a `weight_scale`. Construction fails if that set does not |
| 26 | match the LTX transformer. E4M3 weights and FP32 scales retain their checkpoint |
| 27 | dtypes when the model moves between CPU and CUDA. |
| 28 | |
| 29 | FP8 requires a PyTorch/CUDA build that provides `torch._scaled_mm`. The |
| 30 | reference environment is PyTorch 2.8 with CUDA 12.8. |
| 31 | |
| 32 | ## FP4 |
| 33 | |
| 34 | Install the optional backend and use `configs/inference.fp4.yaml`: |
| 35 | |
| 36 | ```bash |
| 37 | uv pip install -r requirements-fp4.txt |
| 38 | python inference.py --config configs/inference.fp4.yaml |
| 39 | ``` |
| 40 | |
| 41 | The FP4 release contains `components.safetensors` and |
| 42 | `transformer_modelopt.pt`. The first file contains the embedding processor, |
| 43 | video/audio VAEs and vocoder. The second contains the packed FP4 DiT and its |
| 44 | ModelOpt graph metadata. |
| 45 | |
| 46 | The loader constructs the DiT topology on the meta device, restores ModelOpt's |
| 47 | LTX-2 QuantLinear graph, and assigns the packed tensors directly. It never |
| 48 | materializes a BF16 DiT and does not require `echo15_full_dmd`. |
| 49 | |
| 50 | The ModelOpt state is executable framework state and must be obtained from a |
| 51 | trusted release source. ModelOpt names the underlying E2M1 block-scaled format |
| 52 | NVFP4; that technical format name does not mean fake quantization. Native FP4 |
| 53 | execution is hardware dependent. Do not substitute the much larger fake-quant |
| 54 | calibration checkpoint for the packed release artifact. |
| 55 | |
| 56 | ## CLI override |
| 57 | |
| 58 | ```bash |
| 59 | python inference.py \ |
| 60 | --config configs/inference.fp8.yaml \ |
| 61 | --checkpoint /models/echo15_fp8 |
| 62 | |
| 63 | python inference.py \ |
| 64 | --config configs/inference.fp4.yaml \ |
| 65 | --checkpoint /models/echo15_fp4 |
| 66 | ``` |
| 67 | |
| 68 | Run metadata records the public checkpoint, precision and exact backend. The |
| 69 | loader does not accept loose base, DMD, or ModelOpt paths. |
| 70 | |
| 71 | ## RTX 5090 with DiT layerwise offload |
| 72 | |
| 73 | The following measurements use the same R2V case on a Windows RTX 5090: 241 |
| 74 | frames at 1280x736, `resident_blocks=0`, `prefetch_blocks=1`, and tiled VAE |
| 75 | decode. Times are in seconds, GPU memory is the peak NVML reading, and RAM is |
| 76 | the peak process RSS. |
| 77 | |
| 78 | | Checkpoint | Cold start | Online R2V | DMD | Tiled VAE | Peak GPU memory | Peak process RAM | |
| 79 | | --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| 80 | | BF16 | 161.55 s | 138.07 s | 115.21 s | 8.68 s | 16.36 GiB | 70.75 GiB | |
| 81 | | FP8 | 163.11 s | 144.90 s | 125.33 s | 8.66 s | 17.17 GiB | 36.26 GiB | |
| 82 | | FP4 standalone v2 | 160.67 s | 144.09 s | 123.67 s | 8.75 s | 16.29 GiB | 26.51 GiB | |
| 83 | |
| 84 | With full DiT offload, activation memory dominates the GPU peak, so all three |
| 85 | precisions remain near 16-17 GiB. The standalone FP4 v2 release is the default |
| 86 | FP4 checkpoint going forward: it preserves generation speed while reducing |
| 87 | peak process RAM to 26.51 GiB. |
| 88 | |
| 89 | ## H20 with resident DiT |
| 90 | |
| 91 | The resident baseline uses one NVIDIA H20D, the same R2V request and output |
| 92 | shape as the RTX 5090 benchmark, no DiT layerwise offload, and tiled VAE decode. |
| 93 | The conditioning bundle is prepared before loading the generator, matching the |
| 94 | recommended online cache-hit path. Each checkpoint processes the request twice |
| 95 | in one server runtime; the second request is the steady-state result after both |
| 96 | the DiT and decoders are resident on CUDA. |
| 97 | |
| 98 | | Checkpoint | Weight load | First R2V | Steady-state R2V | DMD | Tiled VAE | Peak GPU memory | Peak process RAM | |
| 99 | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
| 100 | | BF16 | 25.33 s | 47.67 s | 44.62 s | 39.36 s | 3.17 s | 47.57 GiB | 38.26 GiB | |
| 101 | | FP8 | 14.65 s | 54.77 s | 51.94 s | 46.42 s | 3.45 s | 33.12 GiB | 20.98 GiB | |
| 102 | | FP4 standalone v2 | 14.12 s | 51.63 s | 48.09 s | 42.50 s | 3.46 s | 25.32 GiB | 16.38 GiB | |
| 103 | |
| 104 | GPU memory is the peak NVML reading during the steady-state request. Process |
| 105 | RAM is the highest sampled RSS during weight loading, which is the lifecycle |
| 106 | peak for these resident runs. The request produced 241-frame, 1280x736 H.264 |
| 107 | video with 48 kHz stereo AAC for all three checkpoints. BF16 is the fastest |
| 108 | resident mode on H20; FP4 provides the lowest resident GPU and host-memory |
| 109 | footprint, while the current FP8 scaled-matmul backend prioritizes memory |
| 110 | reduction rather than latency. |
| 111 |