| 1 | <p align="center"> |
| 2 | <img src="assets/teaser.png" alt="JoyAI-Echo generated video gallery" width="100%"> |
| 3 | </p> |
| 4 | |
| 5 | <div align="center"> |
| 6 | |
| 7 | <h1>JoyAI-Echo</h1> |
| 8 | |
| 9 | <p><strong>🎬 Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds</strong></p> |
| 10 | |
| 11 | <p> |
| 12 | <a href="https://www.researchgate.net/publication/405770309_JoyAI-Echo_Pushing_the_Frontier_of_Long_Audio-Visual_Generation"><b>📄 Paper 1.0</b></a> | |
| 13 | <a href="https://arxiv.org/abs/2608.23383"><b>📄 Paper 1.5</b></a> | |
| 14 | <a href="https://arxiv.org/abs/2608.23189"><b>📄 Echo-WM Paper</b></a> | |
| 15 | <a href="https://arxiv.org/abs/2609.03557"><b>📄 UE Pipeline</b></a> | |
| 16 | <a href="https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/"><b>🌐 Project Page</b></a> | |
| 17 | <a href="https://huggingface.co/jdopensource/JoyAI-Echo"><b>🤗 Long Video Hugging Face</b></a> |
| 18 | </p> |
| 19 | <p> |
| 20 | <a href="https://huggingface.co/Echo-Team/Echo-WM"><b>🤗 World Model Hugging Face</b></a> | |
| 21 | <a href="https://github.com/zhuang2002/ComfyUI_JoyAI_Echo"><b>🖥️ ComfyUI</b></a> |
| 22 | </p> |
| 23 | |
| 24 | </div> |
| 25 | |
| 26 | ## 📰 News |
| 27 | |
| 28 | - 🎮 **2026-09-04** — Released [UE simulation pipeline](https://arxiv.org/abs/2609.03557) for the Echo-WM world data engine, covering physics-based trajectory generation, Movie Render Queue rendering, and distributed scheduling. |
| 29 | - 🚀 **2026-08-28** — Released [JoyAI-Echo 1.5 / Echo-LongVideo](echo_longvideo/README.md), including long-horizon generation, consumer-GPU inference profiles, and the Director Agent. |
| 30 | - 🌍 **2026-08-26** — Released [Echo-WM](echo_wm/README.md), our omnimodal world model for interactive audio-visual generation. |
| 31 | - 🎬 **2026-06-22** — Released [JoyAI-Echo 1.0](https://github.com/jd-opensource/JoyAI-Echo/tree/echo1.0), now preserved on the `echo1.0` archive branch. |
| 32 | |
| 33 | This repository holds two independent projects. Each has its own environment, |
| 34 | checkpoints, and entrypoint — pick the one you need and follow its README. |
| 35 | |
| 36 | | Project | What it does | Guide | |
| 37 | |---|---|---| |
| 38 | | **Echo-LongVideo** (long video) | Long-horizon, multi-shot audio-visual generation. Supports 10+ minutes of long-horizon generation, with a paired audio-video memory bank carrying continuity across shots. | [`echo_longvideo/`](echo_longvideo/README.md) | |
| 39 | | **Echo-WM** (world model) | Omnimodal world model for generative media that responds to continuous navigation while video, environmental sound, music, and speech evolve together. | [`echo_wm/`](echo_wm/README.md) | |
| 40 | |
| 41 | ```text |
| 42 | JoyAI-Echo/ |
| 43 | ├── echo_longvideo/ # long-video generation: inference.py, configs/, prompts/, ltx-* |
| 44 | └── echo_wm/ # world model: inference_wm.py, Gradio demo, bundled ltx-* |
| 45 | ``` |
| 46 | |
| 47 | The two do not share a Python environment or a checkpoint directory. `echo_wm/` |
| 48 | bundles its own copy of `ltx-core` and `ltx-pipelines`, so installing one project |
| 49 | never affects the other. |
| 50 | |
| 51 | ## Quickstart |
| 52 | |
| 53 | Long video: |
| 54 | |
| 55 | ```bash |
| 56 | cd echo_longvideo |
| 57 | conda env create -f environment.yml && conda activate echo-long |
| 58 | ``` |
| 59 | |
| 60 | World model: |
| 61 | |
| 62 | ```bash |
| 63 | cd echo_wm |
| 64 | conda create -n echo-wm python=3.11 -y && conda activate echo-wm |
| 65 | pip install -r requirements.txt |
| 66 | ``` |
| 67 | |
| 68 | Checkpoints are downloaded separately in both cases. See each README for the |
| 69 | exact files and paths. |
| 70 | |
| 71 | **For academic research and non-commercial use only.** |
| 72 | |
| 73 | ## Echo-WM Roadmap |
| 74 | |
| 75 | Echo-WM is on **LTX-2.3** today. Next we move Base and Causal onto **LTX-2.5**, |
| 76 | then cut long-rollout cost with sparse attention and a tighter cache / runtime |
| 77 | stack. |
| 78 | |
| 79 | ### Backbone |
| 80 | |
| 81 | - [x] **LTX-2.3 · Base** — bidirectional audio-visual DiT used by Echo-WM Base (~10 s). |
| 82 | - [x] **LTX-2.3 · Flash Preview / Causal** — current public preview with chunk-causal attention, KV-cache rollout, and 4-step inference. See [`echo_wm/README_CAUSAL.md`](echo_wm/README_CAUSAL.md). |
| 83 | - [ ] **LTX-2.5 · Base** — load official LTX-2.5 weights (Gemma 4 TE, 2.5 VAE / DiT) into the existing bidirectional path. |
| 84 | - [ ] **LTX-2.5 · Causal** — the same Flash recipe on 2.5: block-causal masks, sink+FIFO cache, few-step student. |
| 85 | |
| 86 | ### Accel |
| 87 | |
| 88 | - [ ] **Sparse attention** — SageAttention and similar sparse / low-bit kernels on video, audio, and UCPE branches. |
| 89 | - [ ] **FlashAttention / FlashInfer** — fused attention for long causal windows without blowing up HBM. |
| 90 | - [ ] **Paged KV-cache** — variable-length cache so rollouts stay bounded; rebase RoPE and UCPE when tokens evict. |
| 91 | - [ ] **FP8 / TensorRT** — compile the DiT forward at lower precision for decode-time throughput. |
| 92 | |
| 93 | ## Citation |
| 94 | |
| 95 | If JoyAI-Echo helps your research or products, please cite: |
| 96 | |
| 97 | ```bibtex |
| 98 | @article{duan2026joyaiecho15, |
| 99 | title = {Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds}, |
| 100 | author = {Duan, Nan and Huang, Haoyang and Jin, Weiyang and Li, Haoran and Li, Yaowei and Li, Yuming and Liu, Yijun and Lu, Xin and Ma, Xiaoxiao and Ma, Yanwen and Su, Yaofeng and Sun, Yilang and Wang, Haoyu and Xue, Zeyue and Zhang, Songchun and Zhuang, Junhao}, |
| 101 | journal = {arXiv preprint arXiv:2608.23383}, |
| 102 | year = {2026}, |
| 103 | eprint = {2608.23383}, |
| 104 | archivePrefix = {arXiv}, |
| 105 | primaryClass = {cs.CV}, |
| 106 | url = {https://arxiv.org/abs/2608.23383} |
| 107 | } |
| 108 | |
| 109 | @article{zhang2026echowm, |
| 110 | title = {EchoWM: Open and Enterable Omnimodal World Models}, |
| 111 | author = {Zhang, Songchun and Li, Yaowei and Zhuang, Junhao and Jin, Weiyang and Wang, Haoyu and Lu, Xin and Sun, Yilang and Zhang, Shiyi and Li, Haoran and Ma, Xiaoxiao and Li, Yuming and Liu, Yijun and Su, Yaofeng and Ma, Yanwen and Wu, Haoyu and Su, Zihan and Ma, Yue and Zhang, Lvmin and Huang, Haoyang and Xue, Zeyue and Rao, Anyi and Duan, Nan}, |
| 112 | journal = {arXiv preprint arXiv:2608.23189}, |
| 113 | year = {2026}, |
| 114 | eprint = {2608.23189}, |
| 115 | archivePrefix = {arXiv}, |
| 116 | primaryClass = {cs.CV}, |
| 117 | url = {https://arxiv.org/abs/2608.23189} |
| 118 | } |
| 119 | |
| 120 | @article{li2026joyai, |
| 121 | title = {JoyAI-Echo: Pushing the Frontier of Long Audio-Visual Generation}, |
| 122 | author = {Li, Haoran and Li, Fredreic and Ma, Shichen and Huang, Jie and Liu, Yijun and Shi, Jiaqi and Ma, Yanwen}, |
| 123 | year = {2026} |
| 124 | } |
| 125 | ``` |
| 126 | |
| 127 | ## License |
| 128 | |
| 129 | This project is based on LTX-2 by Lightricks Ltd. |
| 130 | |
| 131 | Portions of the original LTX-2 codebase have been modified by JD.com for academic and research purposes only. |
| 132 | This project is not intended for commercial use. For commercial use of LTX-2 or its derivatives, please contact Lightricks Ltd. |
| 133 | |
| 134 | All original copyright, license, patent, trademark, and attribution notices from LTX-2 are retained. |
| 135 | This project remains subject to the LTX-2 Community License Agreement. |
| 136 |