| 1 | <p align="center"> |
| 2 | <img src="assets/image.png" alt="JoyAI-Echo generated video gallery" width="100%"> |
| 3 | </p> |
| 4 | |
| 5 | <div align="center"> |
| 6 | |
| 7 | <h1>JoyAI-Echo</h1> |
| 8 | |
| 9 | <p><strong>🎬 Pushing the Frontier of Long Video Generation</strong></p> |
| 10 | |
| 11 | <p>Standalone, inference-only release for <strong>minute-level multi-shot audio-video generation</strong> with a distilled DMD generator, paired cross-modal memory, and story-level consistency.</p> |
| 12 | |
| 13 | <p> |
| 14 | <a href="https://www.researchgate.net/publication/405770309_JoyAI-Echo_Pushing_the_Frontier_of_Long_Audio-Visual_Generation"><b>📄 Paper</b></a> | |
| 15 | <a href="https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/"><b>🌐 Project Page</b></a> | |
| 16 | <a href="#quickstart"><b>🚀 Quickstart</b></a> | |
| 17 | <a href="https://huggingface.co/jdopensource/JoyAI-Echo"><b>🤗 Hugging Face</b></a> | |
| 18 | <a href="#results"><b>📊 Results</b></a> | |
| 19 | <a href="https://github.com/zhuang2002/ComfyUI_JoyAI_Echo"><b>🖥️ ComfyUI</b></a> | |
| 20 | <a href="#citation"><b>📝 Citation</b></a> |
| 21 | </p> |
| 22 | |
| 23 | <p> |
| 24 | <img src="https://img.shields.io/badge/Python-3.11-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.11"> |
| 25 | <img src="https://img.shields.io/badge/PyTorch-2.8-EE4C2C?style=flat-square&logo=pytorch&logoColor=white" alt="PyTorch 2.8"> |
| 26 | <img src="https://img.shields.io/badge/CUDA-12.8-76B900?style=flat-square&logo=nvidia&logoColor=white" alt="CUDA 12.8"> |
| 27 | <img src="https://img.shields.io/badge/Release-Inference--Only-black?style=flat-square" alt="Inference"> |
| 28 | <img src="https://img.shields.io/badge/Long%20Video-5%20min-d61f2c?style=flat-square" alt="5 minute long video"> |
| 29 | </p> |
| 30 | |
| 31 | </div> |
| 32 | |
| 33 | ## Abstract |
| 34 | |
| 35 | Long video generation still suffers from error accumulation, weak temporal coherence, and prohibitive latency, limiting its applicability to interactive scenarios. We present **JoyAI-Echo**, a framework that breaks these barriers through four key advances. |
| 36 | Central to its performance, a cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently over five-minute videos, while a post-training pipeline combines memory-based reinforcement learning with distribution matching distillation for a **7.5× speedup** to substantially boost visual quality and alignment. |
| 37 | Empowered by these two components, **JoyAI-Echo** decisively outperforms *HappyOyster* (directing mode) on long-form generation and even surpasses the short-video specialist *Wan 2.6* on human-centric tasks. |
| 38 | Beyond raw generation quality, an interactive agent enables real-time user editing through conversational instructions, and a lightweight super-resolution module maintains high definition under streaming latency, further elevating the overall experience and delivering instantly editable, conversation-speed video creation. |
| 39 | For the first time, **JoyAI-Echo** simultaneously achieves long-range cross-modal consistency, real-time inference for minute-long video, conversational interactivity, and high-resolution output — without compromise, inaugurating a new era of interactive video generation. |
| 40 | Codes and weights will be open-sourced. |
| 41 | |
| 42 | ## Highlights |
| 43 | |
| 44 | - 🎞️ **Minute-level multi-shot stories**: generate a sequence of coherent shots from one prompt JSON. |
| 45 | - ⚡ **DMD-distilled few-step inference**: ~7.5x faster than the original pipeline. |
| 46 | - 🔊 **Joint audio-video generation**: one pipeline produces synchronized video and audio. |
| 47 | - 🧠 **Paired cross-modal memory bank**: conditions each new shot on prior visual identity and voice context for story-level consistency. |
| 48 | |
| 49 | |
| 50 | ## ComfyUI Integration |
| 51 | |
| 52 | Recommended ComfyUI node package: **[ComfyUI_JoyAI_Echo](https://github.com/zhuang2002/ComfyUI_JoyAI_Echo)** — faithful to the official inference pipeline with full bf16 precision (no GGUF quantization), per-shot editable prompts with instant video preview, 3-phase GPU memory hot-swap (48GB VRAM), built-in LLM prompt enhancement, and cross-shot memory chaining for story-level consistency. |
| 53 | |
| 54 | ## Current Release Scope |
| 55 | |
| 56 | JoyAI-Echo currently focuses on **text-to-video (T2V)** and **multi-shot long-video generation with paired audio-video memory**. The memory used in our official pipeline is built from generated T2V shots. |
| 57 | |
| 58 | Please note that **image-to-video (I2V)** is **not supported in the current release**. |
| 59 | |
| 60 | We are actively working on I2V support and plan to release it in a future version. |
| 61 | |
| 62 | ## Demo Gallery |
| 63 | |
| 64 | Explore long-form and short-form JoyAI-Echo cases on the [Project Page](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/). 🍿 |
| 65 | |
| 66 | ## Results |
| 67 | |
| 68 | ### Reported Scale |
| 69 | |
| 70 | | Item | Value | |
| 71 | | --- | ---: | |
| 72 | | 🎬 Long-form coherent story length | **5 min** | |
| 73 | | ⚡ Generation speedup over the original multi-step pipeline | **7.5x** | |
| 74 | | 📚 Benchmark stories | **100** | |
| 75 | | 🎞️ Generated evaluation shots | **3,000** | |
| 76 | | 🕒 Frames per shot | **241 @ 25 fps** | |
| 77 | |
| 78 | ### Human Evaluation |
| 79 | |
| 80 | GSB user study on long- and short-video generation. The numbers denote the percentage of user preferences. |
| 81 | |
| 82 | | Aspect<br>(Long Video) | JoyAI-Echo | Tie | HappyOyster<br> (Directing) | |
| 83 | | --- | ---: | ---: | ---: | |
| 84 | | Visual aesthetics | **63.6%** | 8.8% | 27.6% | |
| 85 | | Audio quality | **81.7%** | 6.5% | 11.8% | |
| 86 | | Prompt following | **80.6%** | 13.5% | 5.9% | |
| 87 | | IP consistency | **59.4%** | 12.9% | 27.7% | |
| 88 | |
| 89 | | Aspect<br>(Short Video) | JoyAI-Echo | Tie | Wan 2.6 | |
| 90 | | --- | ---: | ---: | ---: | |
| 91 | | Visual aesthetics | **58.8%** | 14.7% | 26.5% | |
| 92 | | Audio quality | 32.3% | 30.9% | 36.8% | |
| 93 | | Prompt following | 33.8% | 36.8% | 29.4% | |
| 94 | |
| 95 | ## Repository Layout |
| 96 | |
| 97 | ```text |
| 98 | . |
| 99 | +-- configs/ |
| 100 | | `-- inference.yaml # all inference parameters (YAML) |
| 101 | +-- checkpoints/ # model weights (download separately) |
| 102 | | +-- echo-longvideo-release.safetensors |
| 103 | | `-- gemma-3-12b/ |
| 104 | +-- prompts/ # multi-shot prompt JSON files |
| 105 | | +-- example_single_shot.json |
| 106 | | `-- example_multi_shot.json |
| 107 | +-- ltx-core/src/ltx_core/ # transformer, VAE, text-encoder building blocks |
| 108 | +-- ltx-pipelines/src/ltx_pipelines/ # sampler and pipeline utilities |
| 109 | +-- ltx-distillation/ |
| 110 | | +-- src/ltx_distillation/ # DMD wrappers, AV pipelines, memory bank, utils |
| 111 | | `-- scripts/multishot_inference_dmd.py |
| 112 | +-- inference.py # main entrypoint (load once, infer all) |
| 113 | +-- requirements.txt |
| 114 | `-- environment.yml |
| 115 | ``` |
| 116 | |
| 117 | ## Quickstart |
| 118 | |
| 119 | ### 1. Clone |
| 120 | |
| 121 | ```bash |
| 122 | |
| 123 | git clone https://github.com/jd-opensource/JoyAI-Echo.git |
| 124 | cd JoyAI-Echo |
| 125 | ``` |
| 126 | |
| 127 | ### 2. Create the environment |
| 128 | |
| 129 | The reference environment is **Python 3.11 + PyTorch 2.8 + CUDA 12.8**. |
| 130 | |
| 131 | With conda: |
| 132 | |
| 133 | ```bash |
| 134 | conda env create -f environment.yml |
| 135 | conda activate echo-long |
| 136 | ``` |
| 137 | |
| 138 | With `uv`: |
| 139 | |
| 140 | ```bash |
| 141 | uv venv --python 3.11 .venv |
| 142 | source .venv/bin/activate |
| 143 | uv pip install --extra-index-url https://download.pytorch.org/whl/cu128 -r requirements.txt |
| 144 | ``` |
| 145 | |
| 146 | [`ffmpeg`](https://ffmpeg.org/download.html) must be available on `PATH` for shot concatenation. The conda recipe includes it. If you use `uv`, install it with your system package manager: |
| 147 | |
| 148 | ```bash |
| 149 | sudo apt install ffmpeg |
| 150 | # macOS: |
| 151 | brew install ffmpeg |
| 152 | ``` |
| 153 | |
| 154 | ### 3. Download checkpoint |
| 155 | |
| 156 | Download the JoyAI-Echo release checkpoint and Gemma text encoder: |
| 157 | |
| 158 | | File | Description | Size | Link | |
| 159 | | --- | --- | --- | --- | |
| 160 | | `echo-longvideo-release.safetensors` | Full model (transformer + VAE + vocoder) | ~46 GB |[`JoyAI-Echo`](https://huggingface.co/jdopensource/JoyAI-Echo) | |
| 161 | | `gemma-3-12b/` | Instruction-tuned model (text encoder) | ~24 GB | [`gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) | |
| 162 | |
| 163 | Place them under `checkpoints/`: |
| 164 | |
| 165 | ```text |
| 166 | checkpoints/ |
| 167 | +-- echo-longvideo-release.safetensors |
| 168 | `-- gemma-3-12b/ |
| 169 | ``` |
| 170 | |
| 171 | ### 4. Write a story prompt |
| 172 | |
| 173 | **Enhance your prompt first.** We provide prompt enhancers — system prompts that expand a short story or idea into well-formed shot prompts: **`prompts/long_story_writer_system_prompt.md`** for long, multi-shot video, and **`prompts/short_story_writer_system_prompt.md`** for single-shot short video. We **strongly recommend** running your input through the matching enhancer before inference; un-enhanced prompts tend to produce noticeably weaker results. |
| 174 | |
| 175 | Create a JSON file under `prompts/`. Each file is a single object with a `prompts` list, where **every string is one complete shot**. A single string produces one shot; multiple strings produce a multi-shot story, with each new shot conditioned on the previous ones through the paired audio-video memory bank. |
| 176 | |
| 177 | Inside each string, write these parts in order: |
| 178 | |
| 179 | | Part | What to describe | |
| 180 | | --- | --- | |
| 181 | | **Roles & Subjects** | Describe the appearance of all visible people, including age, build, hair, face, wardrobe, and speaking voice timbre when applicable. | |
| 182 | | **Action & Dialogue** | What the subject does and speaks. | |
| 183 | | **Style** | The overall visual and emotional aesthetic — e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. | |
| 184 | | **Camera Movement** | The shot type and framing or movement — e.g. a stable close-up on the face, or a medium shot from the waist up. | |
| 185 | | **Background** | The setting and scene details behind the subject. | |
| 186 | | **Sound Effects & BGM** | The sounds in the scene and the background music — e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or nobackground music | |
| 187 | |
| 188 | A more convenient prompt-writing workflow will be released as a **director agent** for everyone to use. |
| 189 | |
| 190 | ### 5. Run inference |
| 191 | |
| 192 | ```bash |
| 193 | python inference.py |
| 194 | ``` |
| 195 | |
| 196 | This loads the model once and processes all prompt files under `prompts/`. |
| 197 | |
| 198 | > 💡 **Note**: The inference pipeline is optimized to run on lower-VRAM |
| 199 | > GPUs. Peak GPU usage is around **46–50 GB**, at the cost of slightly |
| 200 | > longer per-shot inference time. |
| 201 | |
| 202 | Outputs are written to: |
| 203 | |
| 204 | ```text |
| 205 | inference_result/outputs/<prompt-name>/inference_<timestamp>/ |
| 206 | ``` |
| 207 | |
| 208 | ## Configuration |
| 209 | |
| 210 | All inference parameters are managed in `configs/inference.yaml`. The file is organized into sections: |
| 211 | |
| 212 | | Section | Contents | |
| 213 | | --- | --- | |
| 214 | | `paths` | Checkpoint path, prompts directory, output root | |
| 215 | | `video` | Resolution, frame count, FPS, seed | |
| 216 | | `denoising` | Step list and sigma schedule | |
| 217 | | `memory` | Memory bank size, save mode, LoRA settings | |
| 218 | | `audio_memory` | Audio window, mel-spectrogram params | |
| 219 | | `inference` | Device, dtype, grad scale | |
| 220 | |
| 221 | ### Override via CLI |
| 222 | |
| 223 | Any YAML parameter can be overridden from the command line: |
| 224 | |
| 225 | ```bash |
| 226 | python inference.py --seed 42 --num-frames 121 |
| 227 | ``` |
| 228 | |
| 229 | Use a custom config file: |
| 230 | |
| 231 | ```bash |
| 232 | python inference.py --config configs/my_experiment.yaml |
| 233 | ``` |
| 234 | |
| 235 | The Python entrypoint exposes the full configuration surface: |
| 236 | |
| 237 | ```bash |
| 238 | python inference.py --help |
| 239 | ``` |
| 240 | |
| 241 | ## Hardware |
| 242 | |
| 243 | Peak GPU usage is around **46–50 GB** for the default **25 fps x 241 frames x 1280 x 736** setting, so a single H100/A100-class (80 GB) or 48 GB GPU is sufficient. |
| 244 | |
| 245 | For smaller GPUs, reduce frames: |
| 246 | |
| 247 | ```bash |
| 248 | python inference.py --num-frames 121 |
| 249 | ``` |
| 250 | |
| 251 | ## TODO List |
| 252 | |
| 253 | - [x] Release inference code |
| 254 | - [x] Release model checkpoints |
| 255 | - [x] Add prompt examples |
| 256 | - [ ] Release Echo-SR (Super-resolution) |
| 257 | - [ ] Release Director Agent |
| 258 | |
| 259 | ## Links |
| 260 | |
| 261 | - Project page: [`https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/`](https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/) |
| 262 | - Repository: [`https://github.com/jd-opensource/JoyAI-Echo`](https://github.com/jd-opensource/JoyAI-Echo) |
| 263 | - Huggingface: [`https://huggingface.co/jdopensource/JoyAI-Echo`](https://huggingface.co/jdopensource/JoyAI-Echo) |
| 264 | - Research Group & Research Paper: [`https://github.com/Echo-Team-Joy-Future-Academy-JD`](https://github.com/Echo-Team-Joy-Future-Academy-JD), [Echo-Memory](https://github.com/Echo-Team-Joy-Future-Academy-JD/Echo-Memory), [Echo-Infinity](https://github.com/Echo-Team-Joy-Future-Academy-JD/Echo-Infinity) |
| 265 | |
| 266 | ## Acknowledgements |
| 267 | |
| 268 | We gratefully acknowledge the open-source projects this work builds upon — in particular [LTX2.3](https://huggingface.co/Lightricks/LTX-2.3) for the base video generator and [Gemma](https://huggingface.co/google/gemma-3-12b-it) for the text encoder. Thanks to the broader research community whose contributions made this release possible. |
| 269 | |
| 270 | **For academic research and non-commercial use only.** |
| 271 | |
| 272 | ## Citation |
| 273 | |
| 274 | If JoyAI-Echo helps your research or products, please cite: |
| 275 | |
| 276 | ```bibtex |
| 277 | @techreport{echo2026joyai, |
| 278 | title = {JoyAI-Echo: Pushing the Frontier of Long Video Generation}, |
| 279 | author = {{Echo Team @ Joy Future Academy, JD}}, |
| 280 | institution = {Joy Future Academy, JD}, |
| 281 | year = {2026}, |
| 282 | month = {May} |
| 283 | } |
| 284 | ``` |
| 285 | |
| 286 | ## License |
| 287 | |
| 288 | This project is based on LTX-2 by Lightricks Ltd. |
| 289 | |
| 290 | Portions of the original LTX-2 codebase have been modified by JD.com for academic and research purposes only. |
| 291 | This project is not intended for commercial use. For commercial use of LTX-2 or its derivatives, please contact Lightricks Ltd. |
| 292 | |
| 293 | All original copyright, license, patent, trademark, and attribution notices from LTX-2 are retained. |
| 294 | This project remains subject to the LTX-2 Community License Agreement. |
| 295 |