README.md
6.5 KB · 152 lines · markdown Raw
1 ---
2 base_model: BreezeBlue/Breeze-TTS-2
3 language:
4 - en
5 - zh
6 library_name: gguf
7 license: other
8 license_name: breezeblue-research-and-non-commercial
9 license_link: https://huggingface.co/BreezeBlue/Breeze-TTS-2
10 pipeline_tag: text-to-speech
11 tags:
12 - text-to-speech
13 - tts
14 - gguf
15 - ggml
16 - vulkan
17 - voice-cloning
18 - voice-conversion
19 ---
20
21 # Breeze-TTS-2 GGUF
22
23 GGUF conversions of [BreezeBlue/Breeze-TTS-2](https://huggingface.co/BreezeBlue/Breeze-TTS-2) for
24 [**Breeze-TTS-2.cpp**](https://github.com/HoppouAI/Breeze-TTS-2.cpp), a C++ reimplementation running on
25 ggml with a Vulkan backend, so it works on NVIDIA, AMD and Intel GPUs and falls back to CPU.
26
27 Bilingual English and Mandarin, 24 kHz, about 2x realtime at Q8_0 on an RTX 3060 and about 3.2x on
28 an RTX 3080, with first audio in under 150 ms.
29
30 These files will not load in llama.cpp. They need the Breeze-TTS-2.cpp runtime, which implements all
31 four stages of the model: the T5Gemma2 text encoder, the Qwen3 backbone, the 15 step depth decoder and
32 the vocoder.
33
34 ## Files
35
36 | File | Size | Notes |
37 | --- | --- | --- |
38 | `breeze-tts-2-f16.gguf` | 5.9 GB | Reference quality, unquantized |
39 | `breeze-tts-2-q8_0.gguf` | 3.3 GB | **Recommended.** No audible loss against F16 |
40 | `breeze-tts-2-q6_k.gguf` | 2.9 GB | |
41 | `breeze-tts-2-q4_k.gguf` | 2.4 GB | Smallest safe choice, holds up well |
42 | `breeze-tts-2-q8_0-dd4.gguf` | 3.2 GB | Experimental, Q8_0 base with a Q4_K depth decoder |
43 | `breeze-tts-2-q8_0-dd2.gguf` | 3.1 GB | Experimental, Q8_0 base with a Q2_K depth decoder |
44 | `breeze-tts-2-q4_k-dd2.gguf` | 2.3 GB | Experimental, Q4_K base with a Q2_K depth decoder |
45
46 Approximate VRAM is about 1 GB above the file size.
47
48 ### About the `-dd` variants
49
50 Everything except the `-dd` files keeps the depth decoder at higher precision than the rest of the
51 model. The `-dd` variants quantize it too, which is why they are smaller.
52
53 The depth decoder runs **15 sequential steps for every single frame of audio**, so on hardware where
54 that is the bottleneck rather than memory bandwidth, shrinking it can speed generation up noticeably.
55 That is the reason these exist and it is worth benchmarking on your own card.
56
57 The tradeoff is that depth codes feed back into the backbone every frame, so quantization error
58 compounds as generation continues. Output holds up early and then drifts progressively muffled and
59 thin past **roughly 45 seconds of continuous generation**. Short lines and dialogue are fine. Long
60 narration is not, and the failure is gradual rather than obvious, so it is easy to miss on quick tests.
61
62 Treat them as experimental. If in doubt, use `q8_0` or `q4_k`.
63
64 ## Performance
65
66 Real time factor for one stream on Vulkan: seconds of audio per second of wall clock, so anything
67 above 1x is faster than playback. `cfg_scale` above 1 turns on guidance, which vocal tags and voice
68 design often need.
69
70 | File | RTX 3060, `cfg_scale` 1 | RTX 3060, `cfg_scale` 3 | RTX 3080, `cfg_scale` 1 | RTX 3080, `cfg_scale` 3 |
71 | --- | --- | --- | --- | --- |
72 | `q4_k` | 2.41x | 1.74x | 3.50x | 2.81x |
73 | `q6_k` | | | 3.67x | 2.87x |
74 | `q8_0-dd4` | 2.32x | 1.69x | 3.72x | 2.83x |
75 | `q8_0` | 1.99x | 1.58x | 3.16x | 2.71x |
76 | `f16` | 1.42x | | 2.71x | 2.32x |
77
78 The RTX 3060 has its tensor core limit unlocked, the RTX 3080 is a stock card. Both are 12 GB.
79 On the 3060 the smaller files are clearly faster, on the 3080 the gap mostly closes.
80
81 The server can also generate several streams at once with `--parallel`. On the RTX 3080, five streams
82 at `q8_0` each ran at 1.46x to 1.68x, and three with guidance at about 1.5x, all with no gaps. The
83 numbers for every model and both cards are in
84 [docs/performance.md](https://github.com/HoppouAI/Breeze-TTS-2.cpp/blob/main/docs/performance.md).
85
86 ## Usage
87
88 ```bash
89 git clone --recursive https://github.com/HoppouAI/Breeze-TTS-2.cpp
90 cd Breeze-TTS-2.cpp
91 cmake -B build -DCMAKE_BUILD_TYPE=Release
92 cmake --build build -j
93 ```
94
95 ```bash
96 # invent a voice from a description
97 build/breeze-cli breeze-tts-2-q8_0.gguf \
98 --text "Welcome aboard. Your journey begins now." \
99 --instruction "A warm, thoughtful young woman with a clear, calm delivery." \
100 --output design.wav
101
102 # clone a voice from a clip
103 build/breeze-cli breeze-tts-2-q8_0.gguf \
104 --text "It is good to hear your voice again." \
105 --ref-audio ref_voice.wav --ref-text "The harbour lights came on one by one as the evening tide began to turn." \
106 --output clone.wav
107 ```
108
109 Or run the server, which has a web UI built in plus HTTP and WebSocket streaming:
110
111 ```bash
112 build/breeze-server breeze-tts-2-q8_0.gguf --host 127.0.0.1 --port 8080 --webui
113 ```
114
115 `ref_voice.wav` in this repo is a sample reference clip. Its transcript is
116 "The harbour lights came on one by one as the evening tide began to turn."
117
118 ## Vocal events
119
120 Inline tags in round brackets produce non speech sounds: `(laugh)`, `(sigh)`, `(cough)`,
121 `(clears throat)`, and `[笑]` or `[叹气]` in Chinese. The vocabulary is free form rather than a fixed
122 token list, so descriptive tags like `(nervous chuckle)` often work.
123
124 They usually need `--cfg-scale 2` to `3` to actually fire. At the default of 1.0 the model treats a tag
125 as a suggestion and tends to read straight past anything outside the common set.
126
127 ## Voice conversion
128
129 The runtime can also respeak an existing recording in a different voice, keeping the original timing,
130 phrasing and emphasis while changing only the speaker. This is not part of the upstream model, it falls
131 out of how the codec separates semantic content from acoustic detail.
132
133 It is experimental. Pitch is regenerated rather than copied, so a converted vocal is re-sung in the
134 target voice's own register instead of at the source's. Whether a tune survives varies clip to clip,
135 and `keep_acoustic 1` or `2` copies the lowest acoustic codebooks from the source to pull more of the
136 original contour through. Judge it by ear, and leave `keep_acoustic` at 0 for ordinary speech.
137
138 ## Conversion
139
140 Produced with `scripts/convert_hf_to_gguf.py` and `breeze-quantize` from the repo. The source download
141 must include the `audio_tokenizer/` directory, which holds the vocoder that the model actually uses at
142 inference time.
143
144 ## License
145
146 Weights are governed by the **BreezeBlue Research and Non-Commercial License** from the
147 [original model](https://huggingface.co/BreezeBlue/Breeze-TTS-2). Converting to GGUF does not change
148 that. The Breeze-TTS-2.cpp source code is Apache 2.0.
149
150 You are responsible for complying with the weight license and for obtaining consent for any reference
151 audio or voices you use.
152