README.md
| 1 | --- |
| 2 | license: other |
| 3 | license_name: boson-higgs-tts-3-research-and-non-commercial-license |
| 4 | license_link: LICENSE |
| 5 | language: |
| 6 | - af |
| 7 | - ar |
| 8 | - as |
| 9 | - ast |
| 10 | - az |
| 11 | - ba |
| 12 | - be |
| 13 | - bg |
| 14 | - bn |
| 15 | - bs |
| 16 | - ca |
| 17 | - ceb |
| 18 | - ckb |
| 19 | - cs |
| 20 | - cy |
| 21 | - da |
| 22 | - de |
| 23 | - el |
| 24 | - en |
| 25 | - eo |
| 26 | - es |
| 27 | - et |
| 28 | - eu |
| 29 | - fa |
| 30 | - fi |
| 31 | - fr |
| 32 | - ga |
| 33 | - gl |
| 34 | - gu |
| 35 | - ha |
| 36 | - he |
| 37 | - hi |
| 38 | - hr |
| 39 | - ht |
| 40 | - hu |
| 41 | - hy |
| 42 | - id |
| 43 | - is |
| 44 | - it |
| 45 | - jv |
| 46 | - ka |
| 47 | - kab |
| 48 | - kam |
| 49 | - kea |
| 50 | - kk |
| 51 | - kn |
| 52 | - ko |
| 53 | - ky |
| 54 | - la |
| 55 | - lb |
| 56 | - lg |
| 57 | - ln |
| 58 | - lt |
| 59 | - luo |
| 60 | - lv |
| 61 | - mhr |
| 62 | - mi |
| 63 | - mk |
| 64 | - ml |
| 65 | - mn |
| 66 | - mr |
| 67 | - ms |
| 68 | - mt |
| 69 | - ne |
| 70 | - nl |
| 71 | - no |
| 72 | - nso |
| 73 | - ny |
| 74 | - oc |
| 75 | - om |
| 76 | - pa |
| 77 | - pl |
| 78 | - ps |
| 79 | - pt |
| 80 | - ro |
| 81 | - ru |
| 82 | - rw |
| 83 | - sd |
| 84 | - sk |
| 85 | - sl |
| 86 | - sn |
| 87 | - so |
| 88 | - sq |
| 89 | - sr |
| 90 | - sv |
| 91 | - sw |
| 92 | - ta |
| 93 | - te |
| 94 | - tg |
| 95 | - tl |
| 96 | - tr |
| 97 | - ug |
| 98 | - uk |
| 99 | - umb |
| 100 | - ur |
| 101 | - uz |
| 102 | - vi |
| 103 | - xh |
| 104 | - zh |
| 105 | - zu |
| 106 | tags: |
| 107 | - text-to-speech |
| 108 | - speech-generation |
| 109 | - voice-agent |
| 110 | - expressive-speech |
| 111 | - controllable-tts |
| 112 | - multilingual-tts |
| 113 | pipeline_tag: text-to-speech |
| 114 | library_name: transformers |
| 115 | --- |
| 116 | |
| 117 | # Higgs TTS 3 |
| 118 | |
| 119 | <div align="left" style="display: flex; justify-content: flex-start; margin-top: 10px;"> <a href="https://www.boson.ai/blog/higgs-audio-v3-tts"> <img src="https://img.shields.io/badge/💚-Boson%20Blog-228B22" style="margin-right: 5px;"> </a> </div> |
| 120 | |
| 121 | Higgs TTS 3 is built for voice chat: it **speaks, not just reads**. It turns model responses into expressive conversational speech across **100+ languages**, with **zero-shot voice cloning** and **inline control** over emotion, style, prosody, pauses, and sound effects. |
| 122 | |
| 123 | > [!TIP] |
| 124 | > Released for research and non-commercial use under the **Boson Higgs TTS 3 Research and Non-Commercial License**. Production, hosted APIs, embedding in a product/service, or reselling the model requires a separate commercial license. Prohibited: voice cloning without consent, impersonation, fraud, election deception, biometric surveillance, or any unlawful use. |
| 125 | |
| 126 | > [!TIP] |
| 127 | > **Free for digital creators — including monetized content.** Under the license's **Creator Use Grant**, creators may use Higgs TTS 3 to make and monetize podcasts, videos, and social posts for free. The one requirement is to credit *Boson AI's Higgs Audio* — either in the audio or prominently in the accompanying text (e.g., the video description or show notes). Suggested credit: *"This audio was created with Boson AI's Higgs Audio — https://www.boson.ai/higgs-audio"*. See the [Creator Use](#creator-use-free-for-digital-creators) section below. |
| 128 | |
| 129 |  |
| 130 | |
| 131 | Higgs autoregressive decoder consumes interleaved text and audio tokens. Audio is encoded by the **Higgs Tokenizer** into 8 codebooks at 25 fps, staggered via a **delay pattern**, then mapped to backbone hidden states through a **multi-codebook fused embedding**. Output codes pass through a **multi-codebook fused head**, are de-delayed, and decoded back to waveform. |
| 132 | |
| 133 | | Component | Spec | |
| 134 | |---|---| |
| 135 | | Backbone | ~4B autoregressive decoder (36 L, hidden=2560, GQA 32/8) | |
| 136 | | Multi-codebook embedding / head | Fused single-tensor, tied with text embedding | |
| 137 | | Context length | 8,192 tokens (training sequence length) | |
| 138 | | Audio tokens | 8 codebooks × 1026 vocab, delay pattern | |
| 139 | | Sample rate | 24 kHz | |
| 140 | | Frame rate | 25 fps (40 ms / frame) | |
| 141 | |
| 142 | ## Supported Languages |
| 143 | |
| 144 | The model reaches **single-digit WER/CER on 102 languages**, which split into two tiers. |
| 145 | |
| 146 | ### WER/CER under 5 — polished, production-quality (85) |
| 147 | |
| 148 | 🇿🇦 Afrikaans · 🇸🇦🇪🇬 Arabic · 🇦🇲 Armenian · 🇮🇳 Assamese · 🇪🇸 Asturian · 🇦🇿 Azerbaijani · 🇷🇺 Bashkir · 🇪🇸 Basque · 🇧🇾 Belarusian · 🇧🇩🇮🇳 Bengali · 🇧🇦 Bosnian · 🇧🇬 Bulgarian · 🇪🇸 Catalan · 🇵🇭 Cebuano · 🇮🇶 Central Kurdish · 🇨🇳 Chinese · 🇭🇷 Croatian · 🇨🇿 Czech · 🇩🇰 Danish · 🇳🇱🇧🇪 Dutch · 🇷🇺 Eastern Mari · 🇺🇸🇬🇧🇦🇺 English · 🌐 Esperanto · 🇪🇪 Estonian · 🇫🇮 Finnish · 🇫🇷🇨🇦 French · 🇪🇸 Galician · 🇬🇪 Georgian · 🇩🇪🇦🇹 German · 🇬🇷 Greek · 🇮🇳 Gujarati · 🇭🇹 Haitian Creole · 🇳🇬 Hausa · 🇮🇱 Hebrew · 🇮🇳 Hindi · 🇭🇺 Hungarian · 🇮🇩 Indonesian · 🇮🇹 Italian · 🇯🇵 Japanese · 🇮🇩 Javanese · 🇮🇳 Kannada · 🇰🇿 Kazakh · 🇰🇷 Korean · 🇷🇼 Kinyarwanda · 🇰🇬 Kyrgyz · 🇱🇻 Latvian · 🇨🇩 Lingala · 🇱🇹 Lithuanian · 🇰🇪 Luo · 🇲🇰 Macedonian · 🇲🇾🇮🇩 Malay · 🇮🇳 Malayalam · 🇲🇹 Maltese · 🇳🇿 Māori · 🇮🇳 Marathi · 🇲🇳 Mongolian · 🇳🇵 Nepali · 🇳🇴 Norwegian · 🇫🇷 Occitan · 🇮🇷🇦🇫 Persian · 🇵🇱 Polish · 🇵🇹🇧🇷 Portuguese · 🇷🇴 Romanian · 🇷🇺 Russian · 🇿🇦 Sepedi · 🇷🇸 Serbian · 🇿🇼 Shona · 🇸🇰 Slovak · 🇸🇮 Slovene · 🇪🇸🇲🇽 Spanish · 🇹🇿🇰🇪 Swahili · 🇸🇪 Swedish · 🇵🇭 Tagalog · 🇹🇯 Tajik · 🇮🇳🇱🇰 Tamil · 🇮🇳 Telugu · 🇹🇭 Thai · 🇹🇷 Turkish · 🇺🇦 Ukrainian · 🇵🇰🇮🇳 Urdu · 🇨🇳 Uyghur · 🇺🇿 Uzbek · 🇻🇳 Vietnamese · 🇿🇦 Xhosa · 🇿🇦 Zulu |
| 149 | |
| 150 | ### WER/CER between 5 and 10 — usable, but less polished (17) |
| 151 | |
| 152 | 🇦🇱 Albanian · 🇲🇼🇿🇲 Chichewa/Nyanja · 🇮🇳🇵🇰 Eastern Punjabi · 🇺🇬 Ganda · 🇮🇸 Icelandic · 🇮🇪 Irish · 🇩🇿 Kabyle · 🇨🇻 Kabuverdianu · 🇰🇪 Kamba · 🇻🇦 Latin · 🇱🇺 Luxembourgish · 🇪🇹🇰🇪 Oromo · 🇦🇫🇵🇰 Pashto · 🇵🇰🇮🇳 Sindhi · 🇸🇴 Somali · 🇦🇴 Umbundu · 🇬🇧 Welsh |
| 153 | |
| 154 | ## Control Tokens |
| 155 | |
| 156 | All tags follow `<|category:value|>` syntax and can be inserted mid-utterance. |
| 157 | |
| 158 | > For how to place these tags when writing the target text (sentence-level vs. inline, `sfx` formatting, stacking, worked examples), see **[PROMPTING.md](./PROMPTING.md)**. |
| 159 | |
| 160 | - **Emotion** — `elation`, `amusement`, `enthusiasm`, `determination`, `pride`, `contentment`, `affection`, `relief`, `contemplation`, `confusion`, `surprise`, `awe`, `longing`, `arousal`, `anger`, `fear`, `disgust`, `bitterness`, `sadness`, `shame`, `helplessness` |
| 161 | |
| 162 | <div style="margin-left:1.5em;margin-top:-10px"> |
| 163 | <table style="display:inline-table;width:max-content;max-width:none;border-collapse:collapse;font-size:clamp(11px,1.4vw,13px);text-align:left"> <thead><tr> <th style="padding:14px 20px;text-align:left;font-weight:600;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Token</th> <th style="padding:14px 20px;text-align:left;font-weight:500;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Description</th> </tr></thead> <tbody> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:elation|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Elation / joy</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:amusement|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Amusement / playful laughter</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:enthusiasm|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Enthusiasm / excitement</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:determination|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Determination / firmness</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:pride|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Pride / confidence</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:contentment|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Calm satisfaction</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:affection|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Warmth / affection</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:relief|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Relief</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:contemplation|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Thoughtful / reflective</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:confusion|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Confused</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:surprise|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Surprised</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:awe|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Awe / wonder</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:longing|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Longing / yearning</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:arousal|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Heightened desire</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:anger|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Anger</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:fear|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Fear</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:disgust|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Disgust</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:bitterness|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Bitterness</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:sadness|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Sadness</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:shame|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Shame</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|emotion:helplessness|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Helplessness</td></tr> </tbody> </table> |
| 164 | </div> |
| 165 | |
| 166 | - **Style** — `singing`, `shouting`, `whispering` |
| 167 | |
| 168 | <div style="margin-left:1.5em;margin-top:-10px"> |
| 169 | <table style="display:inline-table;width:max-content;max-width:none;border-collapse:collapse;font-size:clamp(11px,1.4vw,13px);text-align:left"> <thead><tr> <th style="padding:14px 20px;text-align:left;font-weight:600;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Token</th> <th style="padding:14px 20px;text-align:left;font-weight:500;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Description</th> </tr></thead> <tbody> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|style:singing|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Singing</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|style:shouting|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Shouting / projected voice</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|style:whispering|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Whisper</td></tr> </tbody> </table> |
| 170 | </div> |
| 171 | |
| 172 | - **Sound effects** — `cough`, `laughter`, `crying`, `screaming`, `burping`, `humming`, `sigh`, `sniff`, `sneeze` |
| 173 | |
| 174 | <div style="margin-left:1.5em;margin-top:-10px"> |
| 175 | <p style="margin:0 0 1px;color:rgba(128,128,128,0.9);font-size:13px;text-align:left">Pair each token with the matching onomatopoeia immediately after it.</p> |
| 176 | <table style="display:inline-table;width:max-content;max-width:none;border-collapse:collapse;font-size:clamp(11px,1.4vw,13px);text-align:left"> <thead><tr> <th style="padding:14px 20px;text-align:left;font-weight:600;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Token</th> <th style="padding:14px 20px;text-align:left;font-weight:500;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Description</th> <th style="padding:14px 20px;text-align:left;font-weight:500;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Suggested onomatopoeia</th> </tr></thead> <tbody> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:cough|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Cough</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Ahem</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:laughter|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Laughter</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Haha / Hehe</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:crying|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Crying</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Boohoo / Sob</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:screaming|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Screaming</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Ahh / Aaah</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:burping|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Burping</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Burp</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:humming|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Humming</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Hmm / Mmm</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:sigh|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Sigh</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Uh / Ahh</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:sniff|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Sniff</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Sff</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|sfx:sneeze|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Sneeze</td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Achoo</td></tr> </tbody> </table> |
| 177 | </div> |
| 178 | |
| 179 | - **Prosody** |
| 180 | - Speed — `speed_very_slow`, `speed_slow`, `speed_fast`, `speed_very_fast` |
| 181 | - Pauses — `pause`, `long_pause` |
| 182 | - Pitch — `pitch_low`, `pitch_high` |
| 183 | - Delivery — `expressive_high`, `expressive_low` |
| 184 | |
| 185 | <div style="margin-left:1.5em;margin-top:-10px"> |
| 186 | <table style="display:inline-table;width:max-content;max-width:none;border-collapse:collapse;font-size:clamp(11px,1.4vw,13px);text-align:left"> <thead><tr> <th style="padding:14px 20px;text-align:left;font-weight:600;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Token</th> <th style="padding:14px 20px;text-align:left;font-weight:500;border-bottom:2px solid #7BCFA3;color:#7BCFA3">Effect</th> </tr></thead> <tbody> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:speed_very_slow|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈0.65× speed</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:speed_slow|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈0.85× speed</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:speed_fast|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈1.2× speed</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:speed_very_fast|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈1.4× speed</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:pitch_low|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈−3 semitones</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:pitch_high|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈+2.5 semitones</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:pause|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈400–700 ms pause</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:long_pause|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">≈700–1500 ms pause</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:expressive_high|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">More expressive delivery</td></tr> <tr><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)"><code><|prosody:expressive_low|></code></td><td style="padding:12px 20px;border-bottom:1px solid rgba(128,128,128,0.15)">Flatter delivery</td></tr> </tbody> </table> |
| 187 | </div> |
| 188 | |
| 189 | ## Evaluation Benchmarks |
| 190 | |
| 191 | ### Multilingual Voice Clone |
| 192 | |
| 193 | We evaluate Higgs TTS 3 on public multilingual TTS suites and our internal 111-language Higgs-Multilingual set, covering both common and lower-resource languages. |
| 194 | |
| 195 | WER / CER (↓, ×100) macro-averaged across each benchmark's language set. Lower is better; **bold** marks the best per row. All numbers are reproducible end-to-end with original metrics and normalization. |
| 196 | |
| 197 | <div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;width:100%;padding:16px 0;overflow-x:auto;-webkit-overflow-scrolling:touch"> |
| 198 | <table style="width:100%;border-collapse:collapse;font-size:clamp(11px,1.4vw,13px);text-align:center"> |
| 199 | <thead><tr> |
| 200 | <th style="padding:18px 24px;text-align:center;font-weight:600;border-bottom:2px solid #16a34a;color:#16a34a">Benchmark</th> |
| 201 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Higgs TTS v2</th> |
| 202 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Higgs TTS 3</th> |
| 203 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Fish Audio S2 Pro</th> |
| 204 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Qwen3-TTS-1.7B</th> |
| 205 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">VibeVoice-7B</th> |
| 206 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">IndexTTS-2</th> |
| 207 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">MiMo-Audio-7B-Instruct</th> |
| 208 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">MOSS-TTS-v1.5</th> |
| 209 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">OmniVoice</th> |
| 210 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">ChatterBox</th> |
| 211 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">FireRedTTS-2</th> |
| 212 | </tr></thead> |
| 213 | <tbody> |
| 214 | <tr> |
| 215 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">SeedTTS</td> |
| 216 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">2.10</td> |
| 217 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">1.11</td> |
| 218 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1.31</td> |
| 219 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1.30</td> |
| 220 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">3.59</td> |
| 221 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1.63</td> |
| 222 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">3.70</td> |
| 223 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1.73</td> |
| 224 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1.21</td> |
| 225 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">17.00</td> |
| 226 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1.72</td> |
| 227 | </tr> |
| 228 | <tr> |
| 229 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">CV3</td> |
| 230 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">21.19</td> |
| 231 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">4.41</td> |
| 232 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">4.60</td> |
| 233 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">7.73</td> |
| 234 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">11.66</td> |
| 235 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">129.26</td> |
| 236 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.55</td> |
| 237 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">6.11</td> |
| 238 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">4.92</td> |
| 239 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">32.62</td> |
| 240 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">19.20</td> |
| 241 | </tr> |
| 242 | <tr> |
| 243 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">MiniMax-Multilingual</td> |
| 244 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">49.86</td> |
| 245 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">2.74</td> |
| 246 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">5.15</td> |
| 247 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">27.41</td> |
| 248 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">8.21</td> |
| 249 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">112.91</td> |
| 250 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">85.67</td> |
| 251 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">3.78</td> |
| 252 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">2.98</td> |
| 253 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">49.30</td> |
| 254 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">12.52</td> |
| 255 | </tr> |
| 256 | <tr> |
| 257 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">Higgs-Multilingual</td> |
| 258 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">52.24</td> |
| 259 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">3.61</td> |
| 260 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">8.68</td> |
| 261 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">97.09</td> |
| 262 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">13.74</td> |
| 263 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">57.71</td> |
| 264 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.61</td> |
| 265 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">21.28</td> |
| 266 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">3.63</td> |
| 267 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">57.52</td> |
| 268 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">33.69</td> |
| 269 | </tr> |
| 270 | </tbody> |
| 271 | </table> |
| 272 | </div> |
| 273 | |
| 274 | ### Emergent TTS |
| 275 | |
| 276 | Win-rate (↑) per category — judge preference vs the BASELINE row; **bold** marks the highest win-rate per column. For a fair comparison, every model shares the same reference audio per prompt, and we run the benchmark text verbatim — no inline control tags inserted. |
| 277 | |
| 278 | <div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;width:100%;padding:16px 0;overflow-x:auto;-webkit-overflow-scrolling:touch"> |
| 279 | <table style="width:100%;border-collapse:collapse;font-size:clamp(11px,1.4vw,13px);text-align:center"> |
| 280 | <thead><tr> |
| 281 | <th style="padding:18px 24px;text-align:center;font-weight:600;border-bottom:2px solid #16a34a;color:#16a34a">Model</th> |
| 282 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Overall ↑</th> |
| 283 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Emotions ↑</th> |
| 284 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Foreign Words ↑</th> |
| 285 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Paralinguistics ↑</th> |
| 286 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Complex Pronunciation ↑</th> |
| 287 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Questions ↑</th> |
| 288 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #16a34a;color:#16a34a;font-size:14px">Syntactic Complexity ↑</th> |
| 289 | </tr></thead> |
| 290 | <tbody> |
| 291 | <tr> |
| 292 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">Higgs TTS 3</td> |
| 293 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">53.65%</td> |
| 294 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">53.75%</td> |
| 295 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">48.75%</td> |
| 296 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">68.57%</td> |
| 297 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">25.10%</td> |
| 298 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">61.43%</td> |
| 299 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">60.71%</td> |
| 300 | </tr> |
| 301 | <tr> |
| 302 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">Fish Audio S2 Pro</td> |
| 303 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.80%</td> |
| 304 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">53.04%</td> |
| 305 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">33.93%</td> |
| 306 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">53.75%</td> |
| 307 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">18.16%</td> |
| 308 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.00%</td> |
| 309 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">45.71%</td> |
| 310 | </tr> |
| 311 | <tr> |
| 312 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">Qwen3-TTS-1.7B</td> |
| 313 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">38.84%</td> |
| 314 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">45.54%</td> |
| 315 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">24.64%</td> |
| 316 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.29%</td> |
| 317 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">30.00%</td> |
| 318 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">53.39%</td> |
| 319 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">34.11%</td> |
| 320 | </tr> |
| 321 | <tr> |
| 322 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">IndexTTS-2</td> |
| 323 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">31.12%</td> |
| 324 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">39.29%</td> |
| 325 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">5.36%</td> |
| 326 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">42.50%</td> |
| 327 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">12.45%</td> |
| 328 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">45.89%</td> |
| 329 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">38.93%</td> |
| 330 | </tr> |
| 331 | <tr> |
| 332 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">MOSS-TTS-v1.5</td> |
| 333 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.89%</td> |
| 334 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">60.54%</td> |
| 335 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.18%</td> |
| 336 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.43%</td> |
| 337 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">11.63%</td> |
| 338 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">53.21%</td> |
| 339 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">47.32%</td> |
| 340 | </tr> |
| 341 | <tr> |
| 342 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">OmniVoice</td> |
| 343 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">40.82%</td> |
| 344 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-weight:600">61.07%</td> |
| 345 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">28.75%</td> |
| 346 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">52.68%</td> |
| 347 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">13.67%</td> |
| 348 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">45.00%</td> |
| 349 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">40.36%</td> |
| 350 | </tr> |
| 351 | </tbody> |
| 352 | </table> |
| 353 | </div> |
| 354 | |
| 355 | ## Usage |
| 356 | |
| 357 | ### SGLang Usage |
| 358 | |
| 359 | Pair the weights in this repo with [**SGLang-Omni**](https://github.com/sgl-project/sglang-omni) — a production serving stack with continuous batching for multi-codebook decoding and the same inline tag controls. The Higgs TTS cookbook walks you through installation, server launch, request examples, and the full API reference. |
| 360 | |
| 361 | See the [Higgs TTS cookbook](https://sgl-project.github.io/sglang-omni/cookbook/higgs_tts.html) for the full details. |
| 362 | |
| 363 | |
| 364 | #### Install and Serve |
| 365 | |
| 366 | ```bash |
| 367 | docker pull lmsysorg/sglang-omni:dev |
| 368 | docker run -it --gpus all --shm-size 32g --ipc host --network host --privileged \ |
| 369 | lmsysorg/sglang-omni:dev /bin/zsh |
| 370 | |
| 371 | git clone git@github.com:sgl-project/sglang-omni.git && cd sglang-omni |
| 372 | uv venv .venv -p 3.12 && source .venv/bin/activate |
| 373 | uv pip install -v -e . |
| 374 | ``` |
| 375 | |
| 376 | ```bash |
| 377 | export HF_TOKEN=hf_xxxxxxxxxxxxxxxx |
| 378 | hf download bosonai/higgs-tts-3-4b |
| 379 | |
| 380 | sgl-omni serve \ |
| 381 | --model-path bosonai/higgs-tts-3-4b \ |
| 382 | --port 8000 |
| 383 | ``` |
| 384 | |
| 385 | #### Zero-shot synthesis |
| 386 | |
| 387 | ```bash |
| 388 | curl -X POST http://localhost:8000/v1/audio/speech \ |
| 389 | -H "Content-Type: application/json" \ |
| 390 | -d '{"input": "Hello, how are you?"}' \ |
| 391 | --output output.wav |
| 392 | ``` |
| 393 | |
| 394 | #### Voice cloning |
| 395 | |
| 396 | Supplying the reference transcript (`text`) materially improves cloning fidelity. |
| 397 | |
| 398 | ```python |
| 399 | import requests |
| 400 | |
| 401 | resp = requests.post( |
| 402 | "http://localhost:8000/v1/audio/speech", |
| 403 | json={ |
| 404 | "input": "Have a nice day and enjoy south california sunshine.", |
| 405 | "references": [{ |
| 406 | "audio_path": "ref.wav", |
| 407 | "text": "Hey, Adam here. Let's create something that feels real, sounds human, and connects every time.", |
| 408 | }], |
| 409 | "temperature": 0.8, "top_k": 50, "max_new_tokens": 1024, |
| 410 | }, |
| 411 | ) |
| 412 | with open("output.wav", "wb") as f: |
| 413 | f.write(resp.content) |
| 414 | ``` |
| 415 | |
| 416 | #### Streaming (Server-Sent Events) |
| 417 | |
| 418 | Set `"stream": true` to receive base64-encoded WAV chunks as the vocoder emits them — sub-second time-to-first-audio. Each event carries `audio.data` (base64 WAV bytes); the terminal event has `finish_reason: "stop"` plus usage metadata. |
| 419 | |
| 420 | ```python |
| 421 | import requests, base64, json |
| 422 | |
| 423 | with requests.post( |
| 424 | "http://localhost:8000/v1/audio/speech", |
| 425 | json={"input": "Get the trust fund to the bank early.", "stream": True}, |
| 426 | stream=True, |
| 427 | ) as resp, open("output.wav", "wb") as f: |
| 428 | for line in resp.iter_lines(): |
| 429 | if not line or not line.startswith(b"data: ") or line == b"data: [DONE]": |
| 430 | continue |
| 431 | event = json.loads(line[6:]) |
| 432 | if event.get("finish_reason") == "stop": |
| 433 | break |
| 434 | audio = event.get("audio") or {} |
| 435 | if audio.get("data"): |
| 436 | f.write(base64.b64decode(audio["data"])) |
| 437 | ``` |
| 438 | |
| 439 | #### Inline control tokens |
| 440 | |
| 441 | Embed `<|emotion:…|>`, `<|style:…|>`, `<|prosody:…|>`, and `<|sfx:…|>` tokens directly in `input`. Two rules: |
| 442 | |
| 443 | 1. **Delivery tokens first.** Emotion, style, and the prosody *speed / pitch / expressive* tokens shape the whole turn — put them at the start of `input`. Positional tokens (`<|prosody:pause|>`, `<|prosody:long_pause|>`, `<|sfx:…|>`) go inline exactly where they fire. |
| 444 | 2. **Pair every `<|sfx:…|>` with its onomatopoeia.** E.g. `<|sfx:laughter|>Haha`, `<|sfx:sigh|>Uh`, `<|sfx:sneeze|>Achoo`. The written sound gives the model the acoustic cue to realize the effect. |
| 445 | |
| 446 | Example — amusement + laughter: |
| 447 | |
| 448 | ```bash |
| 449 | curl -X POST http://localhost:8000/v1/audio/speech \ |
| 450 | -H "Content-Type: application/json" \ |
| 451 | -d '{"input": "<|emotion:amusement|><|prosody:expressive_high|>Wait, wait, that was kind of hilarious. <|sfx:laughter|>Hehe, no, seriously, I was not ready for that."}' \ |
| 452 | --output output.wav |
| 453 | ``` |
| 454 | |
| 455 | #### Throughput |
| 456 | |
| 457 | Throughput on Seed-TTS EN (full set, **N=1088** per run). Client `--max-concurrency` sweep against a Higgs server (`max_running_requests=16`, bf16, CUDA Graph on). Each row is the **mean of 3 runs**. Hardware: **1× H100**. |
| 458 | |
| 459 | <div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;width:100%;padding:16px 0;overflow-x:auto;-webkit-overflow-scrolling:touch;text-align:center"> |
| 460 | <table style="display:inline-table;width:max-content;max-width:none;border-collapse:collapse;font-size:clamp(11px,1.4vw,13px);text-align:center"> |
| 461 | <thead><tr> |
| 462 | <th style="padding:18px 24px;text-align:center;font-weight:600;border-bottom:2px solid #ea580c;color:#ea580c">Concurrency</th> |
| 463 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #ea580c;color:#ea580c;font-size:14px">Throughput (req/s)</th> |
| 464 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #ea580c;color:#ea580c;font-size:14px">Mean latency</th> |
| 465 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #ea580c;color:#ea580c;font-size:14px">RTF (per-req)</th> |
| 466 | <th style="padding:18px 24px;text-align:center;font-weight:500;white-space:nowrap;border-bottom:2px solid #ea580c;color:#ea580c;font-size:14px">audio_s/s</th> |
| 467 | </tr></thead> |
| 468 | <tbody> |
| 469 | <tr> |
| 470 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1</td> |
| 471 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1.62</td> |
| 472 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">617 ms</td> |
| 473 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">0.147</td> |
| 474 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">6.89</td> |
| 475 | </tr> |
| 476 | <tr> |
| 477 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">2</td> |
| 478 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">2.70</td> |
| 479 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">742 ms</td> |
| 480 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">0.180</td> |
| 481 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">11.37</td> |
| 482 | </tr> |
| 483 | <tr> |
| 484 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">4</td> |
| 485 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">5.45</td> |
| 486 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">733 ms</td> |
| 487 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">0.177</td> |
| 488 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">22.84</td> |
| 489 | </tr> |
| 490 | <tr> |
| 491 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">8</td> |
| 492 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">8.91</td> |
| 493 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">898 ms</td> |
| 494 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">0.217</td> |
| 495 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">37.38</td> |
| 496 | </tr> |
| 497 | <tr> |
| 498 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">16</td> |
| 499 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">14.74</td> |
| 500 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">1079 ms</td> |
| 501 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">0.262</td> |
| 502 | <td style="padding:16px 24px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">61.84</td> |
| 503 | </tr> |
| 504 | </tbody> |
| 505 | </table> |
| 506 | </div> |
| 507 | |
| 508 | - **Concurrency** — Maximum number of in-flight client requests (`--max-concurrency`). |
| 509 | - **Throughput (req/s)** — Completed requests divided by total benchmark wall-clock time. |
| 510 | - **Mean latency** — Average end-to-end time per request (send to full response received). |
| 511 | - **RTF (per-req)** — Average ratio of processing time to generated audio duration per request (<1 is faster than real time). |
| 512 | - **audio_s/s** — Total seconds of audio produced divided by total benchmark wall-clock time. |
| 513 | |
| 514 | To reproduce the results, follow the instructions in [this script](https://github.com/sgl-project/sglang-omni/blob/main/benchmarks/eval/benchmark_tts_seedtts.py). |
| 515 | |
| 516 | ### vLLM-Omni Usage |
| 517 | |
| 518 | You can also serve these weights with [**vLLM-Omni**](https://github.com/vllm-project/vllm-omni), which exposes the same OpenAI-compatible `/v1/audio/speech` API with zero-shot voice cloning. |
| 519 | |
| 520 | ```bash |
| 521 | hf download bosonai/higgs-tts-3-4b |
| 522 | |
| 523 | vllm-omni serve bosonai/higgs-tts-3-4b \ |
| 524 | --host 0.0.0.0 --port 8095 \ |
| 525 | --trust-remote-code --omni |
| 526 | ``` |
| 527 | |
| 528 | ```bash |
| 529 | curl -X POST http://localhost:8095/v1/audio/speech \ |
| 530 | -H "Content-Type: application/json" \ |
| 531 | -d '{"model": "bosonai/higgs-tts-3-4b", "input": "Hello, how are you?"}' \ |
| 532 | --output output.wav |
| 533 | ``` |
| 534 | |
| 535 | Plain-text TTS, voice-clone, and benchmark recipes are in the [vLLM-Omni Higgs TTS 3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BosonAI/Higgs-Audio-V3-TTS.md). |
| 536 | |
| 537 | ### API Usage |
| 538 | |
| 539 | For zero-ops deployment, use the [**Boson AI API**](https://docs.boson.ai/models/higgs-audio-tts/overview). |
| 540 | |
| 541 | ## Citation |
| 542 | |
| 543 | ```bibtex |
| 544 | @misc{bosonai_higgs_audio_tts_v3_2026, |
| 545 | title = {Higgs TTS 3: Conversational Speech for Voice AI from Boson AI}, |
| 546 | author = {Boson AI}, |
| 547 | year = {2026}, |
| 548 | howpublished = {https://huggingface.co/bosonai/higgs-tts-3-4b}, |
| 549 | } |
| 550 | ``` |
| 551 | |
| 552 | ## Creator Use (Free for Digital Creators) |
| 553 | |
| 554 | In addition to research and non-commercial use, the license includes a **Creator Use Grant** that lets **digital creators** use Higgs TTS 3 to produce creative content **for free — including monetized content**. |
| 555 | |
| 556 | **What's covered** |
| 557 | |
| 558 | - Podcasts, videos, audiobooks, social media posts, and similar creative works |
| 559 | - Personal *and* commercial/monetized creator channels (ad-supported, sponsored, subscription, etc.) |
| 560 | |
| 561 | **The one requirement: acknowledge Boson AI's Higgs Audio.** The acknowledgment must appear in **at least one** of the following ways: |
| 562 | |
| 563 | - **In the audio** — e.g. *"This audio was created with Boson AI's Higgs Audio."* |
| 564 | - **In the accompanying text**, displayed prominently — e.g. in the post body, video description, or show notes. It must be clearly visible and **not** hidden at the bottom of the credits or annotations. |
| 565 | |
| 566 | Suggested credit string: |
| 567 | |
| 568 | > This audio was created with Boson AI's Higgs Audio — https://www.boson.ai/higgs-audio |
| 569 | |
| 570 | **Still requires a separate commercial license.** The Creator Use Grant covers *creating content* with the model. It does **not** cover hosting the model behind an API or as a service, redistributing/reselling/fine-tuning the model for resale, or embedding the model in a product or application. For these uses, [contact us](https://www.boson.ai) for a commercial license. |
| 571 | |
| 572 | The Creator Use Grant does not change the [use restrictions](#license) — no non-consensual voice cloning or impersonation, no fraud or deception, and AI-generated audio must be disclosed where required. Full terms are in Section II-A of the [LICENSE](./LICENSE). |
| 573 | |
| 574 | ## License |
| 575 | |
| 576 | Boson Higgs TTS 3 Research and Non-Commercial License — see [LICENSE](./LICENSE). Includes a **Creator Use Grant** (free monetized creator use with attribution; see [Creator Use](#creator-use-free-for-digital-creators) above). |
| 577 | |