| 1 | <p align="center"> |
| 2 | <img src="assets/image.png" alt="JoyAI-Echo generated video gallery" width="100%"> |
| 3 | </p> |
| 4 | |
| 5 | <div align="center"> |
| 6 | |
| 7 | <h1>JoyAI-Echo 1.5</h1> |
| 8 | |
| 9 | <p><strong>🎬 Long-form audio-video generation with reference-driven multi-shot memory</strong></p> |
| 10 | |
| 11 | <p> |
| 12 | <a href="https://arxiv.org/abs/2608.23383"><b>📄 Paper 1.5</b></a> | |
| 13 | <a href="https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/"><b>🌐 Project Page</b></a> | |
| 14 | <a href="#quickstart"><b>🚀 Quickstart</b></a> | |
| 15 | <a href="https://huggingface.co/jdopensource/JoyAI-Echo"><b>🤗 Model Weights</b></a> | |
| 16 | <a href="Director_Agent/README.md"><b>🎬 Director Agent</b></a> |
| 17 | </p> |
| 18 | |
| 19 | <p> |
| 20 | <img src="https://img.shields.io/badge/Python-3.11-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.11"> |
| 21 | <img src="https://img.shields.io/badge/PyTorch-2.8-EE4C2C?style=flat-square&logo=pytorch&logoColor=white" alt="PyTorch 2.8"> |
| 22 | <img src="https://img.shields.io/badge/CUDA-12.8-76B900?style=flat-square&logo=nvidia&logoColor=white" alt="CUDA 12.8"> |
| 23 | <img src="https://img.shields.io/badge/Release-Inference--Only-black?style=flat-square" alt="Inference only"> |
| 24 | </p> |
| 25 | |
| 26 | </div> |
| 27 | |
| 28 | ## 📢 Whats New |
| 29 | |
| 30 | - 🎉 **JoyAI-Echo 1.5 is now available!** Code, model weights, and R2V inference are released. |
| 31 | - 🎬 **We also introduce Echo Director Agent This Time !**, an agentic workflow for planning and creating multi-shot videos. See [`Director_Agent/`](Director_Agent/README.md). |
| 32 | - 🎮 **JoyAI-Echo 1.5 brings high-quality video generation to consumer GPUs. Fire up your RTX GPU and start creating!** |
| 33 | - 📦 JoyAI-Echo 1.0 remains available on the [`echo1.0` archive branch](https://github.com/jd-opensource/JoyAI-Echo/tree/echo1.0). |
| 34 | |
| 35 | ## JoyAI-Echo 1.5 |
| 36 | |
| 37 | JoyAI-Echo 1.5 is an inference-only release for coherent, long-form |
| 38 | audio-visual generation. It combines few-step generation with a paired |
| 39 | cross-modal memory bank, carrying character appearance, voice, and scene |
| 40 | continuity across independently editable shots. |
| 41 | |
| 42 | |
| 43 | |
| 44 | ## Reference-to-video generation |
| 45 | |
| 46 | Echo 1.5 supports reference-to-video (R2V) generation. Each request may |
| 47 | include a text prompt, an optional first-frame condition, and up to seven |
| 48 | ordered memory slots containing reference images and audio. This makes visual |
| 49 | identity, voice, and story context directly reusable from shot to shot. |
| 50 | |
| 51 | A complete request schema and portable example are available in |
| 52 | [`schemas/r2v_request.schema.json`](schemas/r2v_request.schema.json) and |
| 53 | [`examples/the_last_visa/`](examples/the_last_visa/). 🕶️🍿 |
| 54 | |
| 55 | ## Echo Director Agent |
| 56 | |
| 57 | We also introduce **Echo Director Agent**, a local-first agent for planning, |
| 58 | generating, reviewing and assembling multi-shot videos. It turns a story idea |
| 59 | into an editable production workflow and submits generation jobs to an Echo 1.5 |
| 60 | service. |
| 61 | |
| 62 | See [`Director_Agent/`](Director_Agent/README.md) for installation and usage. |
| 63 | |
| 64 | ## Quickstart |
| 65 | |
| 66 | ### 1. Clone |
| 67 | |
| 68 | ```bash |
| 69 | git clone https://github.com/jd-opensource/JoyAI-Echo.git |
| 70 | cd JoyAI-Echo/echo_longvideo |
| 71 | ``` |
| 72 | |
| 73 | All commands below are run from `echo_longvideo/`. |
| 74 | |
| 75 | ### 2. Install |
| 76 | |
| 77 | The reference environment is Python 3.11, PyTorch 2.8 and CUDA 12.8. We |
| 78 | recommend using [`uv`](https://docs.astral.sh/uv/): |
| 79 | |
| 80 | ```bash |
| 81 | uv venv --python 3.11 .venv |
| 82 | source .venv/bin/activate |
| 83 | uv pip install --extra-index-url https://download.pytorch.org/whl/cu128 \ |
| 84 | -r requirements.txt |
| 85 | python scripts/setup_msst.py |
| 86 | ``` |
| 87 | |
| 88 | [`ffmpeg`](https://ffmpeg.org/download.html) must also be available on `PATH`. |
| 89 | For FP4 inference, install the optional NVIDIA ModelOpt dependency: |
| 90 | |
| 91 | ```bash |
| 92 | uv pip install -r requirements-fp4.txt |
| 93 | ``` |
| 94 | |
| 95 | Conda users can instead run: |
| 96 | |
| 97 | ```bash |
| 98 | conda env create -f environment.yml |
| 99 | conda activate joyai-echo15 |
| 100 | python scripts/setup_msst.py |
| 101 | ``` |
| 102 | |
| 103 | ### 3. Download model weights |
| 104 | |
| 105 | Download the release weights from |
| 106 | [Hugging Face](https://huggingface.co/jdopensource/JoyAI-Echo) and arrange them |
| 107 | as follows: |
| 108 | |
| 109 | | Checkpoint | Precision | Download | |
| 110 | | --- | --- | --- | |
| 111 | | `echo15_full_dmd` | BF16 | [Hugging Face](https://huggingface.co/jdopensource/JoyAI-Echo/tree/main/echo15_full_dmd) | |
| 112 | | `echo15_fp8` | FP8 | [Hugging Face](https://huggingface.co/jdopensource/JoyAI-Echo/tree/main/echo15_fp8) | |
| 113 | | `echo15_fp4` | FP4 | [Hugging Face](https://huggingface.co/jdopensource/JoyAI-Echo/tree/main/echo15_fp4) | |
| 114 | | `gemma-3-12b` | Text encoder | [Hugging Face](https://huggingface.co/google/gemma-3-12b-it) | |
| 115 | |
| 116 | ```text |
| 117 | checkpoints/ |
| 118 | ├── echo15_full_dmd/ # BF16 reference checkpoint |
| 119 | ├── echo15_fp8/ # FP8 scaled-matmul checkpoint |
| 120 | ├── echo15_fp4/ # packed ModelOpt FP4 checkpoint |
| 121 | ├── gemma-3-12b/ # Gemma text encoder |
| 122 | └── msst/ # installed by scripts/setup_msst.py |
| 123 | ``` |
| 124 | |
| 125 | Each Echo checkpoint directory includes its own `checkpoint.json` manifest. |
| 126 | See [`checkpoints/README.md`](checkpoints/README.md) for the exact layout. |
| 127 | |
| 128 | ### 4. Run batch inference |
| 129 | |
| 130 | The default command loads the model once and processes all R2V JSON requests |
| 131 | under `examples/the_last_visa/requests/`: |
| 132 | |
| 133 | ```bash |
| 134 | python inference.py --config configs/inference.bf16.yaml |
| 135 | ``` |
| 136 | |
| 137 | Use FP8 or FP4 by selecting the corresponding configuration: |
| 138 | |
| 139 | ```bash |
| 140 | python inference.py --config configs/inference.fp8.yaml |
| 141 | python inference.py --config configs/inference.fp4.yaml |
| 142 | ``` |
| 143 | |
| 144 | Outputs are written to `inference_result/<work-id>/<shot-id>/`. |
| 145 | |
| 146 | ## Consumer GPU support |
| 147 | |
| 148 | Echo 1.5 includes low-memory profiles for consumer GPUs. They combine |
| 149 | layer-wise DiT weight streaming with tiled Video VAE decoding, trading some |
| 150 | latency for substantially lower peak VRAM usage. |
| 151 | |
| 152 | ```bash |
| 153 | # BF16 (requires substantial system RAM) |
| 154 | python inference.py --config configs/inference.consumer.bf16.yaml |
| 155 | |
| 156 | # FP8 |
| 157 | python inference.py --config configs/inference.consumer.fp8.yaml |
| 158 | |
| 159 | # FP4 standalone (recommended) |
| 160 | python inference.py --config configs/inference.consumer.fp4.yaml |
| 161 | ``` |
| 162 | |
| 163 | For a 24 GiB VRAM target, precompute conditioning before online generation. |
| 164 | Actual headroom depends on the GPU, driver and request shape. |
| 165 | |
| 166 | ## Local inference server |
| 167 | |
| 168 | The repository includes a small local server for Echo Director Agent and other |
| 169 | R2V clients. The root `server.py` is its command-line entry point; the service |
| 170 | implementation lives in the `server/` package. It provides an in-memory queue |
| 171 | and dynamically keeps model weights on the GPU when memory permits. |
| 172 | |
| 173 | ```bash |
| 174 | uv pip install -r requirements-server.txt |
| 175 | uv run python server.py --config configs/server.consumer.yaml |
| 176 | ``` |
| 177 | |
| 178 | On Linux or macOS, after setup, start it with one command: |
| 179 | |
| 180 | ```bash |
| 181 | ./scripts/start_server.sh |
| 182 | ``` |
| 183 | |
| 184 | On Windows, start it from Command Prompt with: |
| 185 | |
| 186 | ```bat |
| 187 | scripts\start_server.cmd |
| 188 | ``` |
| 189 | |
| 190 | Extra server arguments may be appended, for example |
| 191 | `./scripts/start_server.sh --port 8222` or |
| 192 | `scripts\start_server.cmd --port 8222`. |
| 193 | |
| 194 | The server YAML owns deployment settings and references a separate inference |
| 195 | YAML, which owns the checkpoint and pipeline settings. |
| 196 | |
| 197 | See [`docs/LOCAL_SERVER.md`](docs/LOCAL_SERVER.md) for deployment options. |
| 198 | |
| 199 | ## Acknowledgements |
| 200 | |
| 201 | We gratefully acknowledge the open-source projects that make this release |
| 202 | possible, especially [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3), |
| 203 | [Gemma](https://huggingface.co/google/gemma-3-12b-it) and |
| 204 | [MSST-WebUI](https://github.com/SUC-DriverOld/MSST-WebUI). See |
| 205 | [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) for details. |
| 206 | |
| 207 | ## Citation |
| 208 | |
| 209 | If JoyAI-Echo helps your research, please cite: |
| 210 | |
| 211 | ```bibtex |
| 212 | @article{duan2026joyaiecho15, |
| 213 | title = {Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds}, |
| 214 | author = {Duan, Nan and Huang, Haoyang and Jin, Weiyang and Li, Haoran and Li, Yaowei and Li, Yuming and Liu, Yijun and Lu, Xin and Ma, Xiaoxiao and Ma, Yanwen and Su, Yaofeng and Sun, Yilang and Wang, Haoyu and Xue, Zeyue and Zhang, Songchun and Zhuang, Junhao}, |
| 215 | journal = {arXiv preprint arXiv:2608.23383}, |
| 216 | year = {2026}, |
| 217 | eprint = {2608.23383}, |
| 218 | archivePrefix = {arXiv}, |
| 219 | primaryClass = {cs.CV}, |
| 220 | url = {https://arxiv.org/abs/2608.23383} |
| 221 | } |
| 222 | ``` |
| 223 | |
| 224 | **For academic research and non-commercial use only.** |
| 225 | |
| 226 | ## License |
| 227 | |
| 228 | This project is based on LTX-2 by Lightricks Ltd. |
| 229 | |
| 230 | Portions of the original LTX-2 codebase have been modified by JD.com for |
| 231 | academic and research purposes only. This project is not intended for |
| 232 | commercial use. For commercial use of LTX-2 or its derivatives, please contact |
| 233 | Lightricks Ltd. |
| 234 | |
| 235 | All original copyright, license, patent, trademark, and attribution notices |
| 236 | from LTX-2 are retained. This project remains subject to the LTX-2 Community |
| 237 | License Agreement. |
| 238 |