| 1 | <div align="center"> |
| 2 | |
| 3 | <h1>Echo-WM: Open and Enterable Omnimodal World Models</h1> |
| 4 | |
| 5 | **An open audio-visual world model for controllable, persistent, and interactive world generation.** |
| 6 | |
| 7 | [🤗 Model Weights](https://huggingface.co/Echo-Team/Echo-WM) · |
| 8 | [🌐 Project Page](https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/wm/) |
| 9 | |
| 10 | </div> |
| 11 | |
| 12 | ## 🌍 Overview |
| 13 | |
| 14 | **Echo-WM** is an omnimodal world model for generative media that responds to continuous navigation while video, environmental sound, music, and speech evolve together. |
| 15 | |
| 16 | ### Public Inference |
| 17 | |
| 18 | | Model | Horizon | Input | Control | Output | Status | |
| 19 | |---|---:|---|---|---|---| |
| 20 | | **Echo-WM Base** | ~10 s | First-frame image + prompt | Action DSL / pure camera control | Video + audio | Available | |
| 21 | | **Echo-WM Flash Preview** | autoregressive preview | First-frame image + prompt | Action DSL / pure camera control | Video + audio | Available | |
| 22 | |
| 23 | Base-model instructions continue below. For the 4-step **Echo-WM Flash |
| 24 | Preview**, DMD-distilled guidance, and bounded cache options, see |
| 25 | **[README_CAUSAL.md](README_CAUSAL.md)**. |
| 26 | |
| 27 | **We’ve just released the short-horizon (preview) version of Echo-WM Flash, with the long-horizon version coming soon.** |
| 28 | |
| 29 | ## 🚀 Environment Setup |
| 30 | |
| 31 | We recommend a dedicated Python environment: |
| 32 | |
| 33 | ```bash |
| 34 | conda create -n echo-wm python=3.11 -y |
| 35 | conda activate echo-wm |
| 36 | ``` |
| 37 | |
| 38 | Install PyTorch for the CUDA version used by your machine, then install the WM |
| 39 | requirements: |
| 40 | |
| 41 | ```bash |
| 42 | pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \ |
| 43 | --index-url https://download.pytorch.org/whl/cu128 |
| 44 | |
| 45 | cd /path/to/JoyAI-Echo/echo_wm |
| 46 | pip install -r requirements.txt |
| 47 | ``` |
| 48 | |
| 49 | Verify CUDA before loading the checkpoint: |
| 50 | |
| 51 | ```bash |
| 52 | python -c "import torch; print('PyTorch:', torch.__version__); print('CUDA available:', torch.cuda.is_available())" |
| 53 | ``` |
| 54 | |
| 55 | ## 📥 Download Checkpoints |
| 56 | |
| 57 | Download the Echo-WM checkpoints and the Gemma 3 text encoder: |
| 58 | |
| 59 | ```bash |
| 60 | cd /path/to/JoyAI-Echo/echo_wm |
| 61 | |
| 62 | hf download Echo-Team/Echo-WM --local-dir checkpoints |
| 63 | hf download google/gemma-3-12b-it-qat-q4_0-unquantized --local-dir checkpoints/gemma-3 |
| 64 | ``` |
| 65 | |
| 66 | Gemma 3 is a gated repository: accept the license on its model page and run |
| 67 | `hf auth login` first. The text encoder runs in bfloat16, so use the |
| 68 | `-qat-q4_0-unquantized` weights above — not the quantized Q4_0 files. |
| 69 | |
| 70 | ## 💻 Inference |
| 71 | |
| 72 | ### Recommended: run a checked-in case |
| 73 | |
| 74 | The checked-in cases include their own `input.png`, six-field prompt, action |
| 75 | string, FOV, and seed. This is the safest way to verify a complete setup: |
| 76 | |
| 77 | ```bash |
| 78 | cd /path/to/JoyAI-Echo/echo_wm |
| 79 | python scripts/run_wm_case.py \ |
| 80 | --case examples/wm_cases/0010 \ |
| 81 | --checkpoint checkpoints/echo-wm-base.safetensors \ |
| 82 | --gemma-path checkpoints/gemma-3 \ |
| 83 | --output-dir outputs/wm_cases |
| 84 | ``` |
| 85 | |
| 86 | The output is: |
| 87 | |
| 88 | ```text |
| 89 | outputs/wm_cases/0010/result.mp4 |
| 90 | outputs/wm_cases/0010/result.json |
| 91 | ``` |
| 92 | |
| 93 | To run the other cases, replace `0010` with any other case name. Use |
| 94 | `--list` to print them all. |
| 95 | |
| 96 | ### Example: image + prompt + action string |
| 97 | |
| 98 | The repository examples are: |
| 99 | |
| 100 | ```text |
| 101 | examples/wm_cases/0004/input.png magic workshop interior |
| 102 | examples/wm_cases/0010/input.png limestone canyon pool |
| 103 | examples/wm_cases/0011/input.png alpine wingsuit flight |
| 104 | examples/wm_cases/0013/input.png meadow cabin, dog from behind |
| 105 | examples/wm_cases/0014/input.png giant piano ridge above clouds |
| 106 | ``` |
| 107 | |
| 108 | For example: |
| 109 | |
| 110 | ```bash |
| 111 | cd /path/to/JoyAI-Echo/echo_wm |
| 112 | python inference_wm.py \ |
| 113 | --image examples/wm_cases/0010/input.png \ |
| 114 | --prompt "Environment: A clear green river runs through a lush fantasy canyon.\n\nCharacter: A solitary adventurer stands on the shore, seen from behind.\n\nStyle: Painterly high-end fantasy environment art.\n\nPerspective: Wide third-person rear view at standing height.\n\nSounds: Water ripples, footsteps, birds, and soft strings.\n\nSpeech: None." \ |
| 115 | --action-str "w-60,a-60,w-60,d-60" \ |
| 116 | --checkpoint checkpoints/echo-wm-base.safetensors \ |
| 117 | --gemma-path checkpoints/gemma-3 \ |
| 118 | --fov-deg 70 \ |
| 119 | --video-cfg 4.0 \ |
| 120 | --audio-cfg 2.0 \ |
| 121 | --output outputs/result.mp4 |
| 122 | ``` |
| 123 | |
| 124 | The prompt must use the six fields described in |
| 125 | [`PROMPT_SKILL.md`](PROMPT_SKILL.md): `Environment`, `Character`, |
| 126 | `Style`, `Perspective`, `Sounds`, and `Speech`. |
| 127 | |
| 128 | ### Optional controls |
| 129 | |
| 130 | ```text |
| 131 | --negative-prompt TEXT Override the configured negative prompt |
| 132 | --auto-fov Estimate FOV with the optional MoGe-2 helper |
| 133 | --fov-deg FLOAT Use an explicit horizontal FOV (default: 70) |
| 134 | --video-cfg FLOAT Video guidance scale (default: 4.0) |
| 135 | --audio-cfg FLOAT Audio guidance scale (default: 2.0) |
| 136 | --no-audio Write a video-only MP4 (audio is on by default) |
| 137 | --no-action-overlay Skip the HUD copy. A separate <name>_action.mp4 with the |
| 138 | WASD/rotation HUD is written by default. |
| 139 | --steps INT Override the default 30 inference steps |
| 140 | --seed INT Set the random seed |
| 141 | ``` |
| 142 | |
| 143 | `--auto-fov` starts `helpers/moge_fov.py` as a subprocess. It is independent |
| 144 | of prompt generation. The public release has no `--auto-prompt` or VLM prompt |
| 145 | helper. |
| 146 | |
| 147 | ## 🎮 Action Control |
| 148 | |
| 149 | Each Action DSL segment is `<keys>-<frames>`, and segments are joined by commas: |
| 150 | |
| 151 | ```text |
| 152 | w/s forward / backward |
| 153 | a/d strafe left / right |
| 154 | i/k pitch up / down |
| 155 | j/l yaw left / right |
| 156 | none hold the camera still |
| 157 | ``` |
| 158 | |
| 159 | Keys can be combined. Example: |
| 160 | |
| 161 | ```text |
| 162 | w-60,wj-60,w-60,d-60 |
| 163 | ``` |
| 164 | |
| 165 | |
| 166 | ## 🖥️ Web Demo |
| 167 | |
| 168 | Launch the full Gradio interface from the WM directory: |
| 169 | |
| 170 | ```bash |
| 171 | cd /path/to/JoyAI-Echo/echo_wm |
| 172 | CHECKPOINT=checkpoints/echo-wm-base.safetensors \ |
| 173 | GEMMA_PATH=checkpoints/gemma-3 \ |
| 174 | PORT=7860 \ |
| 175 | ./run_gradio.sh |
| 176 | ``` |
| 177 | |
| 178 | Open `http://localhost:7860`. The interface accepts a first-frame image, a |
| 179 | six-field prompt, an Action DSL string, FOV/action settings, and separate |
| 180 | video/audio CFG controls. Audio and the action HUD overlay are both enabled by |
| 181 | default; each has a checkbox to turn it off. |
| 182 | |
| 183 | ## 🚦 Multi-GPU Runs |
| 184 | |
| 185 | Run the checked-in cases across the GPUs you have. Cases are split round-robin, |
| 186 | one inference process per GPU at a time, so the GPU count need not match the |
| 187 | case count: |
| 188 | |
| 189 | ```bash |
| 190 | cd /path/to/JoyAI-Echo/echo_wm |
| 191 | GPU_LIST=0,1,2 bash scripts/run_wm_cases_multigpu.sh |
| 192 | CASES="0010 0014" GPU_LIST=0 bash scripts/run_wm_cases_multigpu.sh # subset, serial |
| 193 | ``` |
| 194 | |
| 195 | ```text |
| 196 | GPU_LIST Comma-separated GPU indices (default: 0,1,2) |
| 197 | CASES Subset of cases to run (default: every case directory found) |
| 198 | PYTHON_BIN Interpreter to use (default: python3) |
| 199 | ACTION_OVERLAY Write the HUD copies (default: 1; set to 0 to skip them) |
| 200 | ``` |
| 201 | |
| 202 | Outputs land in `outputs/wm_cases_multigpu/<case>/` with a per-case |
| 203 | `run_gpu<N>.log`. Failures are reported at the end, after the remaining cases |
| 204 | finish. Don't raise the per-GPU concurrency: loading a checkpoint is |
| 205 | host-memory hungry, and parallel loads can trip a container memory limit even |
| 206 | when device memory is fine. |
| 207 | |
| 208 | ## 📄 Citation |
| 209 | |
| 210 | ```bibtex |
| 211 | @article{zhang2026echowm, |
| 212 | title = {EchoWM: Open and Enterable Omnimodal World Models}, |
| 213 | author = {Zhang, Songchun and Li, Yaowei and Zhuang, Junhao and Jin, Weiyang and Wang, Haoyu and Lu, Xin and Sun, Yilang and Zhang, Shiyi and Li, Haoran and Ma, Xiaoxiao and Li, Yuming and Liu, Yijun and Su, Yaofeng and Ma, Yanwen and Wu, Haoyu and Su, Zihan and Ma, Yue and Zhang, Lvmin and Huang, Haoyang and Xue, Zeyue and Rao, Anyi and Duan, Nan}, |
| 214 | journal = {arXiv preprint arXiv:2608.23189}, |
| 215 | year = {2026}, |
| 216 | eprint = {2608.23189}, |
| 217 | archivePrefix = {arXiv}, |
| 218 | primaryClass = {cs.CV}, |
| 219 | url = {https://arxiv.org/abs/2608.23189} |
| 220 | } |
| 221 | ``` |
| 222 |