| 1 | <div align="center"> |
| 2 | |
| 3 | <h1>Echo-WM Flash Preview: 4-Step Autoregressive Inference</h1> |
| 4 | |
| 5 | [🤗 Model Weights](https://huggingface.co/Echo-Team/Echo-WM) · |
| 6 | [🌐 Project Page](https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/wm/) |
| 7 | |
| 8 | </div> |
| 9 | |
| 10 | ## 🌍 Overview |
| 11 | |
| 12 | **Echo-WM Flash Preview** is a 4-step autoregressive variant of Echo-WM. It |
| 13 | takes a first-frame image, a text prompt, and an Action DSL string, then |
| 14 | generates video and audio jointly. Classifier-free guidance is distilled into |
| 15 | the model through DMD, so inference does not require a separate CFG scale. |
| 16 | |
| 17 | | Model | Denoising | Context cache | Status | |
| 18 | | --- | ---: | --- | --- | |
| 19 | | **Echo-WM Flash Preview** | 4 steps per block | Bounded sink-plus-FIFO | Available | |
| 20 | |
| 21 | The default cache combines a persistent attention sink with recent FIFO |
| 22 | history. Camera conditioning uses bounded anchor translation to preserve |
| 23 | relative geometry as the active window moves. |
| 24 | |
| 25 | The longer-horizon Echo-WM Flash checkpoint will be released alongside a |
| 26 | streaming online demo in a future update. Stay tuned. |
| 27 | |
| 28 | ## 📥 Download Checkpoints |
| 29 | |
| 30 | Download the Echo-WM checkpoints from the `echo_wm` directory: |
| 31 | |
| 32 | ```bash |
| 33 | hf download Echo-Team/Echo-WM --local-dir checkpoints |
| 34 | ``` |
| 35 | |
| 36 | This creates: |
| 37 | |
| 38 | ```text |
| 39 | checkpoints/echo-wm-base.safetensors |
| 40 | checkpoints/echo-wm-flash.safetensors |
| 41 | ``` |
| 42 | |
| 43 | Echo-WM Flash Preview loads the single merged `echo-wm-flash.safetensors` |
| 44 | checkpoint; no separate action adapter or training checkpoint is required. |
| 45 | |
| 46 | Download the Gemma 3 text encoder separately: |
| 47 | |
| 48 | ```bash |
| 49 | hf download google/gemma-3-12b-it-qat-q4_0-unquantized \ |
| 50 | --local-dir checkpoints/gemma-3 |
| 51 | ``` |
| 52 | |
| 53 | Gemma 3 is a gated repository. Accept its license and run `hf auth login` |
| 54 | before downloading. Use the unquantized weights above rather than the quantized |
| 55 | Q4_0 files; the text encoder runs in bfloat16. |
| 56 | |
| 57 | ## 🧪 Validate the Installation |
| 58 | |
| 59 | Check the CLI and list the bundled examples without loading the model: |
| 60 | |
| 61 | ```bash |
| 62 | python inference_wm_causal.py --help |
| 63 | python scripts/run_wm_case_causal.py --list |
| 64 | ``` |
| 65 | |
| 66 | Run the focused tests: |
| 67 | |
| 68 | ```bash |
| 69 | python -m pytest -q tests/test_echo_wm_causal.py |
| 70 | ``` |
| 71 | |
| 72 | ## 💻 Inference |
| 73 | |
| 74 | ### Recommended: run a checked-in case |
| 75 | |
| 76 | Each case contains an input image, the original WBench caption, an Action DSL |
| 77 | string, FOV, and seed. The runner infers the output length from the action |
| 78 | durations: |
| 79 | |
| 80 | ```bash |
| 81 | cd /path/to/JoyAI-Echo/echo_wm |
| 82 | python scripts/run_wm_case_causal.py \ |
| 83 | --case examples/wm_causal_cases/0079 \ |
| 84 | --checkpoint checkpoints/echo-wm-flash.safetensors \ |
| 85 | --gemma-path checkpoints/gemma-3 \ |
| 86 | --video_local_attn_size 19 \ |
| 87 | --video_sink_size 7 \ |
| 88 | --video_chunk_size 3 \ |
| 89 | --output-dir outputs/wm_cases_causal |
| 90 | ``` |
| 91 | |
| 92 | For case `0079`, the four 96-frame actions produce 385 output frames, including |
| 93 | the first frame. Use `--dry-run` to print the resolved command without loading |
| 94 | the model. By default, each run writes both the raw video and a second video |
| 95 | with the action HUD overlay, named `<output_stem>_action.mp4`. Pass |
| 96 | `--no-action-overlay` to skip the overlay copy. |
| 97 | |
| 98 | ### Example: image + prompt + action string |
| 99 | |
| 100 | ```bash |
| 101 | python inference_wm_causal.py \ |
| 102 | --image examples/wm_causal_cases/0079/input.jpg \ |
| 103 | --prompt "An enchanted crystal cave with massive prismatic crystal formations in purple, teal, and pink. Bioluminescent fungi glow on the cave floor and walls. Floating light motes drift through the air. The crystals refract light into rainbow spectra. To the right, a large crystalline cave monster with glowing purple eyes lurks behind tall crystal clusters. Deep cavern atmosphere with ethereal luminescence. Further to the right beyond the monster, a subterranean crystal pool glows with turquoise light, fed by a thin waterfall dripping from a stalactite cluster. The cave opens into a wider chamber with an ancient stone altar covered in glowing runes. First-person viewer. First-person view with the right hand holding a twisted wooden magic wand topped with a bright blue-white crystal orb that radiates light. The wand rotates together with the viewer's perspective when turning." \ |
| 104 | --action-str "l-96,l-96,l-96,l-96" \ |
| 105 | --checkpoint checkpoints/echo-wm-flash.safetensors \ |
| 106 | --gemma-path checkpoints/gemma-3 \ |
| 107 | --video_local_attn_size 19 \ |
| 108 | --video_sink_size 7 \ |
| 109 | --video_chunk_size 3 \ |
| 110 | --num-frames 385 \ |
| 111 | --fov-deg 70 \ |
| 112 | --output outputs/flash_result.mp4 |
| 113 | ``` |
| 114 | |
| 115 | The checked-in causal examples keep the original WBench captions verbatim. |
| 116 | They are passed directly to the Gemma tokenizer/text encoder; causal inference |
| 117 | does not invoke a prompt enhancer or rewrite them into six fields. |
| 118 | |
| 119 | ### Optional controls |
| 120 | |
| 121 | ```text |
| 122 | --video_local_attn_size INT Total video cache window (default: 19) |
| 123 | --video_sink_size INT Persistent video sink frames (default: 7) |
| 124 | --video_chunk_size INT Generated video frames per latent block (Flash Preview: 3) |
| 125 | --num-frames INT Decoded output length (default: 241) |
| 126 | --no-audio Write a video-only MP4 |
| 127 | --action-overlay Write a separate action HUD copy (default: enabled) |
| 128 | --no-action-overlay Skip the action HUD copy |
| 129 | --seed INT Set the random seed |
| 130 | ``` |
| 131 | |
| 132 | Hyphenated aliases such as `--video-local-attn-size` are accepted. The checked-in |
| 133 | 16-second examples use 385 frames rather than the direct CLI default of 241. |
| 134 | The corresponding audio local-attention window and sink size are derived |
| 135 | automatically from the video settings to keep audio and video blocks aligned. |
| 136 | |
| 137 | ## 🎮 Action Control |
| 138 | |
| 139 | Each Action DSL segment is `<keys>-<frames>`, and segments are joined by commas: |
| 140 | |
| 141 | ```text |
| 142 | w/s forward / backward |
| 143 | a/d strafe left / right |
| 144 | i/k pitch up / down |
| 145 | j/l yaw left / right |
| 146 | none hold the camera still |
| 147 | ``` |
| 148 | |
| 149 | Keys can be combined, for example `w-96,wj-96,w-96,d-96`. |
| 150 | |
| 151 | ## 🖼️ Example Gallery |
| 152 | |
| 153 | | Case | Scene | Four 4-second actions | Action DSL | |
| 154 | | -------: | ------------------------ | ---------------------------------------- | --------------------------- | |
| 155 | | `0024` | Ancient Roman ruins | Yaw right/forward alternating | `l-96,w-96,l-96,w-96` | |
| 156 | | `0075` | Cherry blossom garden | Forward four times | `w-96,w-96,w-96,w-96` | |
| 157 | | `0079` | Enchanted crystal cave | Yaw right four times | `l-96,l-96,l-96,l-96` | |
| 158 | | `0081` | Sunlit artist studio | Pitch down/up, backward/forward | `k-96,i-96,s-96,w-96` | |
| 159 | | `0122` | Volcanic crater | Strafe left/right alternating | `a-96,d-96,a-96,d-96` | |
| 160 | | `0170` | Mythological marble hall | Forward/backward, strafe left, yaw right | `w-96,s-96,a-96,l-96` | |
| 161 | |
| 162 | The images and original WBench captions are under |
| 163 | [`examples/wm_causal_cases`](examples/wm_causal_cases). All cases use the |
| 164 | WBench `wbench_4s_turn_rot0.4_trans0.05` camera setup. |
| 165 | |
| 166 | ### Multi-GPU batch inference |
| 167 | |
| 168 | Run all checked-in cases across the selected GPUs: |
| 169 | |
| 170 | ```bash |
| 171 | GPU_LIST=0,1,2 bash scripts/run_wm_causal_cases_multigpu.sh |
| 172 | ``` |
| 173 | |
| 174 | Cases are assigned round-robin, with at most one inference process active per |
| 175 | GPU. Select a subset with: |
| 176 | |
| 177 | ```bash |
| 178 | CASES="0081 0170" GPU_LIST=0,1 bash scripts/run_wm_causal_cases_multigpu.sh |
| 179 | ``` |
| 180 | |
| 181 | Supported environment variables: |
| 182 | |
| 183 | ```text |
| 184 | GPU_LIST Comma-separated GPU indices (default: 0,1,2) |
| 185 | CASES Subset of cases (default: every checked-in case) |
| 186 | PYTHON_BIN Interpreter to use (default: python from the active environment) |
| 187 | ACTION_OVERLAY Write action HUD copies (default: 1; set to 0 to skip them) |
| 188 | ``` |
| 189 | |
| 190 | Outputs are written under `outputs/wm_causal_cases_multigpu/<case>/`. Each |
| 191 | successful case contains `result.mp4`, `result_action.mp4`, and a per-case |
| 192 | `run_gpu<N>.log`. Set `ACTION_OVERLAY=0` to omit `result_action.mp4`. |
| 193 | |
| 194 | ## ⚙️ 4-Step Configuration |
| 195 | |
| 196 | Public defaults are in |
| 197 | [`configs/inference_wm_causal.yaml`](configs/inference_wm_causal.yaml): |
| 198 | |
| 199 | ```yaml |
| 200 | video: |
| 201 | width: 1280 |
| 202 | height: 704 |
| 203 | num_frames: 241 |
| 204 | fps: 24 |
| 205 | |
| 206 | causal: |
| 207 | timesteps: [1000, 750, 500, 250] |
| 208 | video_local_attn_size: 19 |
| 209 | video_sink_size: 7 |
| 210 | video_chunk_size: 3 |
| 211 | |
| 212 | action: |
| 213 | translation_speed: 0.05 |
| 214 | rotation_speed_deg: 0.4 |
| 215 | pitch_limit_deg: 40.0 |
| 216 | fov_deg: 70.0 |
| 217 | ``` |
| 218 | |
| 219 | The three causal window parameters are measured in video latent frames. |
| 220 | Echo-WM Flash Preview uses a fixed `video_chunk_size` of 3. Each transformer layer |
| 221 | maintains five bounded temporal caches: video self-attention, audio |
| 222 | self-attention, audio-to-video cross-attention, video-to-audio cross-attention, |
| 223 | and UCPE camera attention. The video settings directly configure the |
| 224 | video-side caches; the aligned audio window and sink are derived automatically |
| 225 | (the defaults map `19/7` video frames to `152/52` audio frames). Video and audio |
| 226 | text K/V are static prompt caches initialized once and are not part of these |
| 227 | five rolling caches. |
| 228 | |
| 229 | `--num-frames` is measured in decoded frames. Valid output lengths follow |
| 230 | `1 + 24m` (for example, 241 or 385 frames), corresponding to `1 + 3m` video |
| 231 | latent frames after the VAE temporal compression. |
| 232 | |
| 233 | ## 📄 Citation |
| 234 | |
| 235 | ```bibtex |
| 236 | @article{echo_wm, |
| 237 | title = {Echo-WM: Open and Enterable Omnimodal World Models}, |
| 238 | author = {Echo Team}, |
| 239 | journal = {arXiv preprint}, |
| 240 | year = {2026} |
| 241 | } |
| 242 | ``` |
| 243 |