| 1 | ## Triton Inference Serving Best Practice for F5-TTS |
| 2 | |
| 3 | ### Setup |
| 4 | #### Option 1: Quick Start |
| 5 | ```sh |
| 6 | # Directly launch the service using docker compose |
| 7 | MODEL=F5TTS_v1_Base docker compose up |
| 8 | ``` |
| 9 | |
| 10 | #### Option 2: Build from scratch |
| 11 | ```sh |
| 12 | # Build the docker image |
| 13 | docker build . -f Dockerfile.server -t soar97/triton-f5-tts:24.12 |
| 14 | |
| 15 | # Create Docker Container |
| 16 | your_mount_dir=/mnt:/mnt |
| 17 | docker run -it --name "f5-server" --gpus all --net host -v $your_mount_dir --shm-size=2g soar97/triton-f5-tts:24.12 |
| 18 | ``` |
| 19 | |
| 20 | ### Build TensorRT-LLM Engines and Launch Server |
| 21 | Inside docker container, we would follow the official guide of TensorRT-LLM to build qwen and whisper TensorRT-LLM engines. See [here](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/models/core/whisper). |
| 22 | ```sh |
| 23 | # F5TTS_v1_Base | F5TTS_Base | F5TTS_v1_Small | F5TTS_Small |
| 24 | bash run.sh 0 4 F5TTS_v1_Base |
| 25 | ``` |
| 26 | > [!NOTE] |
| 27 | > If use custom checkpoint, set `ckpt_file` and `vocab_file` in `run.sh`. |
| 28 | > Remember to used matched model version (`F5TTS_v1_*` for v1, `F5TTS_*` for v0). |
| 29 | > |
| 30 | > If use checkpoint of different structure, see `scripts/convert_checkpoint.py`, and perform modification if necessary. |
| 31 | |
| 32 | > [!IMPORTANT] |
| 33 | > If train or finetune with fp32, add `--dtype float32` flag when converting checkpoint in `run.sh` phase 1. |
| 34 | |
| 35 | ### HTTP Client |
| 36 | ```sh |
| 37 | python3 client_http.py |
| 38 | ``` |
| 39 | |
| 40 | ### Benchmarking |
| 41 | #### Using Client-Server Mode |
| 42 | ```sh |
| 43 | # bash run.sh 5 5 F5TTS_v1_Base |
| 44 | num_task=2 |
| 45 | python3 client_grpc.py --num-tasks $num_task --huggingface-dataset yuekai/seed_tts --split-name wenetspeech4tts |
| 46 | ``` |
| 47 | |
| 48 | #### Using Offline TRT-LLM Mode |
| 49 | ```sh |
| 50 | # bash run.sh 7 7 F5TTS_v1_Base |
| 51 | batch_size=1 |
| 52 | split_name=wenetspeech4tts |
| 53 | backend_type=trt |
| 54 | log_dir=./tests/benchmark_batch_size_${batch_size}_${split_name}_${backend_type} |
| 55 | rm -r $log_dir |
| 56 | torchrun --nproc_per_node=1 \ |
| 57 | benchmark.py --output-dir $log_dir \ |
| 58 | --batch-size $batch_size \ |
| 59 | --enable-warmup \ |
| 60 | --split-name $split_name \ |
| 61 | --model-path $ckpt_file \ |
| 62 | --vocab-file $vocab_file \ |
| 63 | --vocoder-trt-engine-path $VOCODER_TRT_ENGINE_PATH \ |
| 64 | --backend-type $backend_type \ |
| 65 | --tllm-model-dir $TRTLLM_ENGINE_DIR || exit 1 |
| 66 | ``` |
| 67 | |
| 68 | ### Benchmark Results |
| 69 | Decoding on a single L20 GPU, using 26 different prompt_audio & target_text pairs, 16 NFE. |
| 70 | |
| 71 | | Model | Concurrency | Avg Latency | RTF | Mode | |
| 72 | |---------------------|----------------|-------------|--------|-----------------| |
| 73 | | F5-TTS Base (Vocos) | 2 | 253 ms | 0.0394 | Client-Server | |
| 74 | | F5-TTS Base (Vocos) | 1 (Batch_size) | - | 0.0402 | Offline TRT-LLM | |
| 75 | | F5-TTS Base (Vocos) | 1 (Batch_size) | - | 0.1467 | Offline Pytorch | |
| 76 | |
| 77 | ### Credits |
| 78 | 1. [Yuekai Zhang](https://github.com/yuekaizhang) |
| 79 | 2. [F5-TTS-TRTLLM](https://github.com/Bigfishering/f5-tts-trtllm) |
| 80 |