| 1 | # Inference |
| 2 | |
| 3 | The pretrained model checkpoints can be reached at [🤗 Hugging Face](https://huggingface.co/SWivid/F5-TTS) and [🤖 Model Scope](https://www.modelscope.cn/models/SWivid/F5-TTS_Emilia-ZH-EN), or will be automatically downloaded when running inference scripts. |
| 4 | |
| 5 | **More checkpoints with whole community efforts can be found in [SHARED.md](SHARED.md), supporting more languages.** |
| 6 | |
| 7 | Currently support **30s for a single** generation, which is the **total length** (same logic if `fix_duration`) including both prompt and output audio. However, `infer_cli` and `infer_gradio` will automatically do chunk generation for longer text. Long reference audio will be **clip short to ~12s**. |
| 8 | |
| 9 | To avoid possible inference failures, make sure you have seen through the following instructions. |
| 10 | |
| 11 | - Use reference audio <12s and leave proper silence space (e.g. 1s) at the end. Otherwise there is a risk of truncating in the middle of word, leading to suboptimal generation. |
| 12 | - <ins>Uppercased letters</ins> (best with form like K.F.C.) will be uttered letter by letter, and lowercased letters used for common words. |
| 13 | - Add some spaces (blank: " ") or punctuations (e.g. "," ".") <ins>to explicitly introduce some pauses</ins>. |
| 14 | - If English punctuation marks the end of a sentence, make sure there is a space " " after it. Otherwise not regarded as when chunk. |
| 15 | - <ins>Preprocess numbers</ins> to Chinese letters if you want to have them read in Chinese, otherwise in English. |
| 16 | - If the generation output is blank (pure silence), <ins>check for FFmpeg installation</ins>. |
| 17 | - Try <ins>turn off `use_ema` if using an early-stage</ins> finetuned checkpoint (which goes just few updates). |
| 18 | |
| 19 | |
| 20 | ## Gradio App |
| 21 | |
| 22 | Currently supported features: |
| 23 | |
| 24 | - Basic TTS with Chunk Inference |
| 25 | - Multi-Style / Multi-Speaker Generation |
| 26 | - Voice Chat powered by Qwen2.5-3B-Instruct |
| 27 | - [Custom inference with more language support](SHARED.md) |
| 28 | |
| 29 | The cli command `f5-tts_infer-gradio` equals to `python src/f5_tts/infer/infer_gradio.py`, which launches a Gradio APP (web interface) for inference. |
| 30 | |
| 31 | The script will load model checkpoints from Huggingface. You can also manually download files and update the path to `load_model()` in `infer_gradio.py`. Currently only load TTS models first, will load ASR model to do transcription if `ref_text` not provided, will load LLM model if use Voice Chat. |
| 32 | |
| 33 | More flags options: |
| 34 | |
| 35 | ```bash |
| 36 | # Automatically launch the interface in the default web browser |
| 37 | f5-tts_infer-gradio --inbrowser |
| 38 | |
| 39 | # Set the root path of the application, if it's not served from the root ("/") of the domain |
| 40 | # For example, if the application is served at "https://example.com/myapp" |
| 41 | f5-tts_infer-gradio --root_path "/myapp" |
| 42 | ``` |
| 43 | |
| 44 | Could also be used as a component for larger application: |
| 45 | ```python |
| 46 | import gradio as gr |
| 47 | from f5_tts.infer.infer_gradio import app |
| 48 | |
| 49 | with gr.Blocks() as main_app: |
| 50 | gr.Markdown("# This is an example of using F5-TTS within a bigger Gradio app") |
| 51 | |
| 52 | # ... other Gradio components |
| 53 | |
| 54 | app.render() |
| 55 | |
| 56 | main_app.launch() |
| 57 | ``` |
| 58 | |
| 59 | |
| 60 | ## CLI Inference |
| 61 | |
| 62 | The cli command `f5-tts_infer-cli` equals to `python src/f5_tts/infer/infer_cli.py`, which is a command line tool for inference. |
| 63 | |
| 64 | The script will load model checkpoints from Huggingface. You can also manually download files and use `--ckpt_file` to specify the model you want to load, or directly update in `infer_cli.py`. |
| 65 | |
| 66 | For change vocab.txt use `--vocab_file` to provide your `vocab.txt` file. |
| 67 | |
| 68 | Basically you can inference with flags: |
| 69 | ```bash |
| 70 | # Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage) |
| 71 | f5-tts_infer-cli \ |
| 72 | --model F5TTS_v1_Base \ |
| 73 | --ref_audio "ref_audio.wav" \ |
| 74 | --ref_text "The content, subtitle or transcription of reference audio." \ |
| 75 | --gen_text "Some text you want TTS model generate for you." |
| 76 | |
| 77 | # Use BigVGAN as vocoder. Currently only support F5TTS_Base. |
| 78 | f5-tts_infer-cli --model F5TTS_Base --vocoder_name bigvgan --load_vocoder_from_local |
| 79 | |
| 80 | # Use custom path checkpoint, e.g. |
| 81 | f5-tts_infer-cli --ckpt_file ckpts/F5TTS_v1_Base/model_1250000.safetensors |
| 82 | |
| 83 | # More instructions |
| 84 | f5-tts_infer-cli --help |
| 85 | ``` |
| 86 | |
| 87 | And a `.toml` file would help with more flexible usage. |
| 88 | |
| 89 | ```bash |
| 90 | f5-tts_infer-cli -c custom.toml |
| 91 | ``` |
| 92 | |
| 93 | For example, you can use `.toml` to pass in variables, refer to `src/f5_tts/infer/examples/basic/basic.toml`: |
| 94 | |
| 95 | ```toml |
| 96 | # F5TTS_v1_Base | E2TTS_Base |
| 97 | model = "F5TTS_v1_Base" |
| 98 | ref_audio = "infer/examples/basic/basic_ref_en.wav" |
| 99 | # If an empty "", transcribes the reference audio automatically. |
| 100 | ref_text = "Some call me nature, others call me mother nature." |
| 101 | gen_text = "I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring." |
| 102 | # File with text to generate. Ignores the text above. |
| 103 | gen_file = "" |
| 104 | remove_silence = false |
| 105 | output_dir = "tests" |
| 106 | ``` |
| 107 | |
| 108 | You can also leverage `.toml` file to do multi-style generation, refer to `src/f5_tts/infer/examples/multi/story.toml`. |
| 109 | |
| 110 | ```toml |
| 111 | # F5TTS_v1_Base | E2TTS_Base |
| 112 | model = "F5TTS_v1_Base" |
| 113 | ref_audio = "infer/examples/multi/main.flac" |
| 114 | # If an empty "", transcribes the reference audio automatically. |
| 115 | ref_text = "" |
| 116 | gen_text = "" |
| 117 | # File with text to generate. Ignores the text above. |
| 118 | gen_file = "infer/examples/multi/story.txt" |
| 119 | remove_silence = true |
| 120 | output_dir = "tests" |
| 121 | |
| 122 | [voices.town] |
| 123 | ref_audio = "infer/examples/multi/town.flac" |
| 124 | ref_text = "" |
| 125 | |
| 126 | [voices.country] |
| 127 | ref_audio = "infer/examples/multi/country.flac" |
| 128 | ref_text = "" |
| 129 | ``` |
| 130 | You should mark the voice with `[main]` `[town]` `[country]` whenever you want to change voice, refer to `src/f5_tts/infer/examples/multi/story.txt`. |
| 131 | |
| 132 | ## API Usage |
| 133 | |
| 134 | ```python |
| 135 | from importlib.resources import files |
| 136 | from f5_tts.api import F5TTS |
| 137 | |
| 138 | f5tts = F5TTS() |
| 139 | wav, sr, spec = f5tts.infer( |
| 140 | ref_file=str(files("f5_tts").joinpath("infer/examples/basic/basic_ref_en.wav")), |
| 141 | ref_text="some call me nature, others call me mother nature.", |
| 142 | gen_text="""I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences.""", |
| 143 | file_wave=str(files("f5_tts").joinpath("../../tests/api_out.wav")), |
| 144 | file_spec=str(files("f5_tts").joinpath("../../tests/api_out.png")), |
| 145 | seed=None, |
| 146 | ) |
| 147 | ``` |
| 148 | Check [api.py](../api.py) for more details. |
| 149 | |
| 150 | ## TensorRT-LLM Deployment |
| 151 | |
| 152 | See [detailed instructions](../runtime/triton_trtllm/README.md) for more information. |
| 153 | |
| 154 | ## Socket Real-time Service |
| 155 | |
| 156 | Real-time voice output with chunk stream: |
| 157 | |
| 158 | ```bash |
| 159 | # Start socket server |
| 160 | python src/f5_tts/socket_server.py |
| 161 | |
| 162 | # If PyAudio not installed |
| 163 | sudo apt-get install portaudio19-dev |
| 164 | pip install pyaudio |
| 165 | |
| 166 | # Communicate with socket client |
| 167 | python src/f5_tts/socket_client.py |
| 168 | ``` |
| 169 | |
| 170 | ## Speech Editing |
| 171 | |
| 172 | To test speech editing capabilities, use the following command: |
| 173 | |
| 174 | ```bash |
| 175 | python src/f5_tts/infer/speech_edit.py |
| 176 | ``` |
| 177 | |
| 178 |