| 1 | |
| 2 | # Evaluation |
| 3 | |
| 4 | Install packages for evaluation: |
| 5 | |
| 6 | ```bash |
| 7 | pip install -e .[eval] |
| 8 | ``` |
| 9 | |
| 10 | > [!IMPORTANT] |
| 11 | > For [faster-whisper](https://github.com/SYSTRAN/faster-whisper), for various compatibilities: |
| 12 | > `pip install ctranslate2==4.5.0` if CUDA 12 and cuDNN 9; |
| 13 | > `pip install ctranslate2==4.4.0` if CUDA 12 and cuDNN 8; |
| 14 | > `pip install ctranslate2==3.24.0` if CUDA 11 and cuDNN 8. |
| 15 | |
| 16 | ## Generating Samples for Evaluation |
| 17 | |
| 18 | ### Prepare Test Datasets |
| 19 | |
| 20 | 1. *Seed-TTS testset*: Download from [seed-tts-eval](https://github.com/BytedanceSpeech/seed-tts-eval). |
| 21 | 2. *LibriSpeech test-clean*: Download from [OpenSLR](http://www.openslr.org/12/). |
| 22 | 3. Unzip the downloaded datasets and place them in the `data/` directory. |
| 23 | 4. Our filtered LibriSpeech-PC 4-10s subset: `data/librispeech_pc_test_clean_cross_sentence.lst` |
| 24 | |
| 25 | ### Batch Inference for Test Set |
| 26 | |
| 27 | To run batch inference for evaluations, execute the following commands: |
| 28 | |
| 29 | ```bash |
| 30 | # if not setup accelerate config yet |
| 31 | accelerate config |
| 32 | |
| 33 | # if only perform inference |
| 34 | bash src/f5_tts/eval/eval_infer_batch.sh --infer-only |
| 35 | |
| 36 | # if inference and with corresponding evaluation, setup the following tools first |
| 37 | bash src/f5_tts/eval/eval_infer_batch.sh |
| 38 | ``` |
| 39 | |
| 40 | ## Objective Evaluation on Generated Results |
| 41 | |
| 42 | ### Download Evaluation Model Checkpoints |
| 43 | |
| 44 | 1. Chinese ASR Model: [Paraformer-zh](https://huggingface.co/funasr/paraformer-zh) |
| 45 | 2. English ASR Model: [Faster-Whisper](https://huggingface.co/Systran/faster-whisper-large-v3) |
| 46 | 3. WavLM Model: Download from [Google Drive](https://drive.google.com/file/d/1-aE1NfzpRCLxA4GUxX9ITI3F9LlbtEGP/view). |
| 47 | |
| 48 | > [!NOTE] |
| 49 | > ASR model will be automatically downloaded if `--local` not set for evaluation scripts. |
| 50 | > Otherwise, you should update the `asr_ckpt_dir` path values in `eval_librispeech_test_clean.py` or `eval_seedtts_testset.py`. |
| 51 | > |
| 52 | > WavLM model must be downloaded and your `wavlm_ckpt_dir` path updated in `eval_librispeech_test_clean.py` and `eval_seedtts_testset.py`. |
| 53 | |
| 54 | ### Objective Evaluation Examples |
| 55 | |
| 56 | Update the path with your batch-inferenced results, and carry out WER / SIM / UTMOS evaluations: |
| 57 | ```bash |
| 58 | # Evaluation [WER] for Seed-TTS test [ZH] set |
| 59 | python src/f5_tts/eval/eval_seedtts_testset.py --eval_task wer --lang zh --gen_wav_dir <GEN_WAV_DIR> --gpu_nums 8 |
| 60 | |
| 61 | # Evaluation [SIM] for LibriSpeech-PC test-clean (cross-sentence) |
| 62 | python src/f5_tts/eval/eval_librispeech_test_clean.py --eval_task sim --gen_wav_dir <GEN_WAV_DIR> --librispeech_test_clean_path <TEST_CLEAN_PATH> |
| 63 | |
| 64 | # Evaluation [UTMOS]. --ext: Audio extension |
| 65 | python src/f5_tts/eval/eval_utmos.py --audio_dir <WAV_DIR> --ext wav |
| 66 | ``` |
| 67 | |
| 68 | > [!NOTE] |
| 69 | > Evaluation results can also be found in `_*_results.jsonl` files saved in `<GEN_WAV_DIR>`/`<WAV_DIR>`. |
| 70 |