README.md · higgs-tts-2-3b-base

1

---

2

license: other

3

language:

4

- en

5

- zh

6

- de

7

- ko

8

pipeline_tag: text-to-speech

9

library_name: transformers

10

---

11

12

# Higgs TTS 2: Redefining Expressiveness in Audio Generation

13

14

<div align="center" style="display: flex; justify-content: center; margin-top: 10px; flex-wrap: wrap; gap: 8px;">

15

  <a href="https://boson.ai/blog/higgs-audio-v2"><img src='https://img.shields.io/badge/🚀-Launch Blogpost-228B22' style="margin-right: 5px;"></a>

16

  <a href="https://github.com/boson-ai/higgs-audio"><img src="https://img.shields.io/badge/💻-Github%20Repo-9C276A" style="margin-right: 5px;"></a>

17

  <a href="https://huggingface.co/spaces/smola/higgs_audio_v2"><img src="https://img.shields.io/badge/🎮-HF%20Space%20Playground-8A2BE2" style="margin-right: 5px;"></a>

18

  <a href="https://huggingface.co/bosonai/higgs-audio-v2-tokenizer"><img src="https://img.shields.io/badge/🎧-Audio%20Tokenizer-6A5ACD.svg" style="margin-right: 5px;"></a>

19

</div>

20

21

Check our open-source repository https://github.com/boson-ai/higgs-audio for more details!

22

23

> **Rename note:** Higgs Audio V2 and Higgs Audio V2 Generation have been renamed to Higgs TTS 2.

24

25

We are open-sourcing Higgs TTS 2, a powerful audio foundation model pretrained on over 10 million hours of audio data and a diverse set of text data.

26

Despite having no post-training or fine-tuning, Higgs TTS 2 excels in expressive audio generation, thanks to its deep language and acoustic understanding.

27

28

On [EmergentTTS-Eval](https://github.com/boson-ai/emergenttts-eval-public), the model achieves win rates of **75.7%** and **55.7%** over "gpt-4o-mini-tts" on the "Emotions" and "Questions" categories, respectively. It also obtains state-of-the-art performance on traditional TTS benchmarks like Seed-TTS Eval and Emotional Speech Dataset (ESD). Moreover, the model demonstrates capabilities rarely seen in previous systems, including automatic prosody adaptation during narration, zero-shot generation of natural multi-speaker dialogues in multiple languages, melodic humming with the cloned voice, and simultaneous generation of speech and background music.

29

30

31

<p>

32

<img src="./emergent-tts-emotions-win-rate.png" width=900>

33

</p>

34

35

Here's the demo video that shows some of its emergent capabilities (remember to unmute):

36

37

<div align="left">

38

<video width="95%" controls>

39

        <source src="https://cdn-uploads.huggingface.co/production/uploads/64fa072a52e82dd432460767/bjbWGg1IKoMtWXnl0Od8G.mp4" type="video/mp4">

40

Your browser does not support the video tag.

41

</video>

42

</div>

43

44

Here's another demo video that show-cases the model's multilingual capability and how it enabled live translation (remember to unmute):

45

46

<div align="left">

47

<video width="95%" controls>

48

        <source src="https://cdn-uploads.huggingface.co/production/uploads/64fa072a52e82dd432460767/9cN-ky02GzmUgogsIh1Wg.mp4" type="video/mp4">

49

Your browser does not support the video tag.

50

</video>

51

</div>

52

53

## Technical Details

54

55

<p>

56

<img src="./higgs_audio_v2_architecture_combined.png" width=900>

57

</p>

58

59

Higgs TTS 2 adopts the "generation variant" depicted in the architecture figure above. Its strong performance is driven by three key technical innovations:

60

61

- We developed an automated annotation pipeline that leverages multiple ASR models, sound event classification models, and our in-house audio understanding model. Using this pipeline, we cleaned and annotated 10 million hours audio data, which we refer to as AudioVerse. The in-house understanding model is finetuned on top of Higgs Audio v1 Understanding, which adopts the "understanding variant" shown in the architecture figure.

62

- We trained a unified audio tokenizer from scratch that captures both semantic and acoustic features.

63

- We proposed the DualFFN architecture, which enhances the LLM’s ability to model acoustics tokens with minimal computational overhead.

64

65

66

### Audio Tokenizer

67

68

<p>

69

<img src="./higgs_audio_tokenizer_architecture.png" width=900>

70

</p>

71

72

We introduce a new discretized audio tokenizer that runs at just 25 frames per second while keeping—or even improving—audio quality compared to tokenizers with twice the bitrate.

73

Our model is the first to train on 24 kHz data covering speech, music, and sound events in one unified system.

74

It also uses a simple non-diffusion encoder/decoder for fast, batch inference. It achieves state-of-the-art performance in semantic and acoustic evaluations.

75

Check https://huggingface.co/bosonai/higgs-audio-v2-tokenizer for more information about the tokenizer.

76

77

### Model Architecture -- Dual FFN

78

79

Higgs TTS 2 is built on top of [Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B). To enhance the model’s ability to process audio tokens,

80

we incorporate the "DualFFN" architecture as an audio adapter.

81

DualFFN acts as an audio-specific expert, boosting the LLM's performance with minimal computational overhead.

82

Our implementation preserves 91% of the original LLM’s training speed with the inclusion of DualFFN, which has 2.2B parameters.

83

Thus, the total number of parameter for Higgs TTS 2 is 3.6B (LLM) + 2.2B (Audio Dual FFN), and it has the same training / inference FLOPs as Llama-3.2-3B.

84

Ablation study shows that the model equipped with DualFFN consistently outperforms its counterpart in terms of word error rate (WER) and speaker similarity.

85

See [our architecture blog](https://github.com/boson-ai/higgs-audio/blob/main/tech_blogs/ARCHITECTURE_BLOG.md) for more information.

86

87

88

## Evaluation

89

90

Here's the performance of Higgs TTS 2 on four benchmarks,  [Seed-TTS Eval](https://github.com/BytedanceSpeech/seed-tts-eval), [Emotional Speech Dataset (ESD)](https://paperswithcode.com/dataset/esd), [EmergentTTS-Eval](https://arxiv.org/abs/2505.23009), and Multi-speaker Eval:

91

92

#### Seed-TTS Eval & ESD

93

94

We prompt Higgs TTS 2 with the reference text, reference audio, and target text for zero-shot TTS. We use the standard evaluation metrics from Seed-TTS Eval and ESD.

95

96

| | SeedTTS-Eval| | ESD | |

97

|------------------------------|--------|--------|---------|-------------------|

98

| | WER ↓ | SIM ↑ | WER ↓ | SIM (emo2vec) ↑ |

99

| Cosyvoice2 | 2.28 | 65.49 | 2.71 | 80.48 |

100

| Qwen2.5-omni† | 2.33 | 64.10 | - | - |

101

| ElevenLabs Multilingual V2 | **1.43** | 50.00 | 1.66 | 65.87 |

102

| Higgs Audio v1 | 2.18 | 66.27 | **1.49** | 82.84 |

103

| Higgs TTS 2 (base) | 2.44 | **67.70** | 1.78 | **86.13** |

104

105

106

#### EmergentTTS-Eval ("Emotions" and "Questions")

107

108

Following the [EmergentTTS-Eval Paper](https://arxiv.org/abs/2505.23009), we report the win-rate over "gpt-4o-mini-tts" with the "alloy" voice. Results of Higgs TTS 2 are obtained with the voice of "belinda". The judge model is Gemini 2.5 Pro.

109

110

| Model | Emotions (%) ↑ | Questions (%) ↑ |

111

|------------------------------------|--------------|----------------|

112

| Higgs TTS 2 (base) | **75.71%** | **55.71%** |

113

| [gpt-4o-audio-preview†](https://platform.openai.com/docs/models/gpt-4o-audio-preview)       | 61.64%       | 47.85%         |

114

| [Hume.AI](https://www.hume.ai/research) | 61.60% | 43.21% |

115

| **BASELINE:** [gpt-4o-mini-tts](https://platform.openai.com/docs/models/gpt-4o-mini-tts)  | 50.00%       | 50.00%         |

116

| [Qwen 2.5 Omni†](https://github.com/QwenLM/Qwen2.5-Omni) | 41.60% | 51.78% |

117

| [minimax/speech-02-hd](https://replicate.com/minimax/speech-02-hd) | 40.86% | 47.32% |

118

| [ElevenLabs Multilingual v2](https://elevenlabs.io/blog/eleven-multilingual-v2)         | 30.35%       | 39.46%         |

119

| [DeepGram Aura-2](https://deepgram.com/learn/introducing-aura-2-enterprise-text-to-speech)                    | 29.28%       | 48.21%         |

120

| [Sesame csm-1B](https://github.com/SesameAILabs/csm) | 15.96% | 31.78% |

121

122

<sup><sub>'†' means using the strong-prompting method described in the paper.</sub></sup>

123

124

125

#### Multi-speaker Eval

126

127

We also designed a multi-speaker evaluation benchmark to evaluate the capability of Higgs TTS 2 for multi-speaker dialog generation. The benchmark contains three subsets

128

129

- `two-speaker-conversation`: 1000 synthetic dialogues involving two speakers. We fix two reference audio clips to evaluate the model's ability in double voice cloning for utterances ranging from 4 to 10 dialogues between two randomly chosen persona.

130

- `small talk (no ref)`: 250 synthetic dialogues curated in the same way as above, but are characterized by short utterances and a limited number of turns (4–6), we do not fix reference audios in this case and this set is designed to evaluate the model's ability to automatically assign appropriate voices to speakers.

131

- `small talk (ref)`: 250 synthetic dialogues similar to above, but contains even shorter utterances as this set is meant to include reference clips in it's context, similar to `two-speaker-conversation`.

132

133

134

We report the word-error-rate (WER) and the geometric mean between intra-speaker similarity and inter-speaker dis-similarity on these three subsets. Other than Higgs TTS 2, we also evaluated [MoonCast](https://github.com/jzq2000/MoonCast) and [nari-labs/Dia-1.6B-0626](https://huggingface.co/nari-labs/Dia-1.6B-0626), two of the most popular open-source models capable of multi-speaker dialog generation.

135

Results are summarized in the following table. We are not able to run [nari-labs/Dia-1.6B-0626](https://huggingface.co/nari-labs/Dia-1.6B-0626) on our "two-speaker-conversation" subset due to its strict limitation on the length of the utterances and output audio.

136

137

|                                                | two-speaker-conversation |                |small talk |                | small talk (no ref) |                |

138

| ---------------------------------------------- | -------------- | ------------------ | ---------- | -------------- | ------------------- | -------------- |

139

|                                                | WER ↓                      | Mean Sim & Dis-sim ↑ | WER ↓       |  Mean Sim & Dis-sim ↑ | WER ↓               | Mean Sim & Dis-sim ↑ |

140

| [MoonCast](https://github.com/jzq2000/MoonCast) | 38.77                    | 46.02         | **8.33**       | 63.68          | 24.65               | 53.94 |

141

| [nari-labs/Dia-1.6B-0626](https://huggingface.co/nari-labs/Dia-1.6B-0626)         | \-                       | \-             | 17.62      | 63.15          | 19.46               | **61.14**          |

142

| Higgs TTS 2 (base)        | **18.88**                    | **51.95**          | 11.89      | **67.92**              | **14.65**               | 55.28              |

143

144

145

## Usage

146

147

### Transformers 🤗

148

149

Higgs TTS 2 is supported natively in `transformers`: [see the doc](https://huggingface.co/docs/transformers/en/model_doc/higgs_audio_v2).

150

151

```bash

152

uv pip install "transformers>=5.3.0"

153

```

154

155

<details>

156

<summary>Single-speaker smart voice</summary>

157

158

```python

159

from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration

160

161

model_id = "bosonai/higgs-audio-v2-generation-3B-base"

162

processor = AutoProcessor.from_pretrained(model_id, device_map="auto")

163

model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")

164

165

conversation = [

166

{

167

"role": "system",

168

"content": [{"type": "text", "text": "Generate audio following instruction."}],

169

},

170

{

171

"role": "scene",

172

"content": [{"type": "text", "text": "Audio is recorded from a quiet room."}],

173

},

174

{

175

"role": "user",

176

"content": [

177

{

178

"type": "text",

179

                "text": "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.",

180

}

181

],

182

},

183

]

184

185

inputs = processor.apply_chat_template(

186

conversation,

187

add_generation_prompt=True,

188

tokenize=True,

189

return_dict=True,

190

sampling_rate=24000,

191

return_tensors="pt",

192

).to(model.device)

193

194

outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)

195

decoded = processor.batch_decode(outputs)

196

processor.save_audio(decoded, "output_single_speaker.wav")

197

```

198

199

</details>

200

201

<details>

202

<summary>Multi-speaker smart voice</summary>

203

204

Use `[SPEAKER*]` tags to generate a multi-speaker dialogue. Speaker characteristics are described in the `scene` role.

205

206

```python

207

from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration

208

209

model_id = "bosonai/higgs-audio-v2-generation-3B-base"

210

processor = AutoProcessor.from_pretrained(model_id, device_map="auto")

211

model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")

212

213

system_message = """You are an AI assistant designed to convert text into speech.

214

If the user's message includes a [SPEAKER*] tag, do not read out the tag and generate speech for the following text, using the specified voice.

215

If no speaker tag is present, select a suitable voice on your own."""

216

217

user_message = """[SPEAKER0] I can't believe you did that without even asking me first!

218

[SPEAKER1] Oh, come on! It wasn't a big deal, and I knew you would overreact like this.

219

[SPEAKER0] Overreact? You made a decision that affects both of us without even considering my opinion!

220

[SPEAKER1] Because I didn't have time to sit around waiting for you to make up your mind! Someone had to act."""

221

222

conversation = [

223

{

224

"role": "system",

225

"content": [{"type": "text", "text": system_message}],

226

},

227

{

228

"role": "scene",

229

"content": [

230

{"type": "text", "text": "Audio is recorded from a quiet room."},

231

{"type": "text", "text": "SPEAKER0: feminine"},

232

{"type": "text", "text": "SPEAKER1: masculine"},

233

],

234

},

235

{

236

"role": "user",

237

"content": [{"type": "text", "text": user_message}],

238

},

239

]

240

241

inputs = processor.apply_chat_template(

242

conversation,

243

add_generation_prompt=True,

244

tokenize=True,

245

return_dict=True,

246

sampling_rate=24000,

247

return_tensors="pt",

248

).to(model.device)

249

250

outputs = model.generate(**inputs, max_new_tokens=2000, do_sample=False)

251

decoded = processor.batch_decode(outputs)

252

processor.save_audio(decoded, "output_multi_speaker.wav")

253

```

254

255

</details>

256

257

<details>

258

<summary>Zero-shot voice cloning</summary>

259

260

Clone a voice by providing a reference audio in the conversation history.

261

262

```python

263

from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration

264

265

model_id = "bosonai/higgs-audio-v2-generation-3B-base"

266

processor = AutoProcessor.from_pretrained(model_id, device_map="auto")

267

model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")

268

269

conversation = [

270

{

271

"role": "system",

272

"content": [{"type": "text", "text": "Generate audio following instruction."}],

273

},

274

{

275

"role": "scene",

276

"content": [{"type": "text", "text": "Audio is recorded from a quiet room."}],

277

},

278

{

279

"role": "user",

280

"content": [

281

{

282

"type": "text",

283

                "text": "It was the night before my birthday. Hooray! It's almost here! It may not be a holiday, but it's the best day of the year.",

284

}

285

],

286

},

287

{

288

"role": "assistant",

289

"content": [

290

{

291

"type": "audio",

292

"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/belinda.wav",

293

}

294

],

295

},

296

{

297

"role": "user",

298

"content": [

299

{

300

"type": "text",

301

                "text": "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.",

302

}

303

],

304

},

305

]

306

307

inputs = processor.apply_chat_template(

308

conversation,

309

add_generation_prompt=True,

310

tokenize=True,

311

return_dict=True,

312

sampling_rate=24000,

313

return_tensors="pt",

314

).to(model.device)

315

316

outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)

317

decoded = processor.batch_decode(outputs)

318

processor.save_audio(decoded, "output_voice_cloning.wav")

319

```

320

321

</details>

322

323

<details>

324

<summary>Multi-speaker voice cloning</summary>

325

326

Clone multiple voices by providing reference audio clips in the `scene` role.

327

328

```python

329

from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration

330

331

model_id = "bosonai/higgs-audio-v2-generation-3B-base"

332

processor = AutoProcessor.from_pretrained(model_id, device_map="auto")

333

model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")

334

335

user_message = """[SPEAKER0] I can't believe you did that without even asking me first!

336

[SPEAKER1] Oh, come on! It wasn't a big deal, and I knew you would overreact like this.

337

[SPEAKER0] Overreact? You made a decision that affects both of us without even considering my opinion!

338

[SPEAKER1] Because I didn't have time to sit around waiting for you to make up your mind! Someone had to act."""

339

340

conversation = [

341

{

342

"role": "system",

343

"content": [{"type": "text", "text": "Generate audio following instruction."}],

344

},

345

{

346

"role": "scene",

347

"content": [

348

{"type": "text", "text": "Audio is recorded from a quiet room."},

349

{"type": "text", "text": "SPEAKER0:"},

350

{

351

"type": "audio",

352

"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/guess_age_gender.wav",

353

},

354

{"type": "text", "text": "SPEAKER1:"},

355

{

356

"type": "audio",

357

"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac",

358

},

359

],

360

},

361

{

362

"role": "user",

363

"content": [{"type": "text", "text": user_message}],

364

},

365

]

366

367

inputs = processor.apply_chat_template(

368

conversation,

369

add_generation_prompt=True,

370

tokenize=True,

371

return_dict=True,

372

sampling_rate=24000,

373

return_tensors="pt",

374

).to(model.device)

375

376

outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)

377

decoded = processor.batch_decode(outputs)

378

processor.save_audio(decoded, "output_multi_speaker_cloning.wav")

379

```

380

381

</details>

382

383

<details>

384

<summary>Batched inference</summary>

385

386

Process multiple conversations in a single forward pass.

387

388

```python

389

from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration

390

391

model_id = "bosonai/higgs-audio-v2-generation-3B-base"

392

processor = AutoProcessor.from_pretrained(model_id, device_map="auto")

393

model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")

394

395

conversation1 = [

396

{"role": "system", "content": [{"type": "text", "text": "Generate audio following instruction."}]},

397

{"role": "scene", "content": [{"type": "text", "text": "Audio is recorded from a quiet room."}]},

398

{

399

"role": "user",

400

"content": [

401

{

402

"type": "text",

403

                "text": "It was the night before my birthday. Hooray! It's almost here! It may not be a holiday, but it's the best day of the year.",

404

}

405

],

406

},

407

{

408

"role": "assistant",

409

"content": [

410

{

411

"type": "audio",

412

"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/belinda.wav",

413

}

414

],

415

},

416

{

417

"role": "user",

418

"content": [

419

{

420

"type": "text",

421

                "text": "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.",

422

}

423

],

424

},

425

]

426

427

conversation2 = [

428

{"role": "system", "content": [{"type": "text", "text": "Generate audio following instruction."}]},

429

{"role": "scene", "content": [{"type": "text", "text": "Audio is recorded from a quiet room."}]},

430

{

431

"role": "user",

432

"content": [

433

{

434

"type": "text",

435

                "text": " It's super important to assess fairly the fact that our former model is over. And this is not a question of adjustment. This is not the same world, 2024, 2025. And on top of that, we are making the same mistakes, on top of the key elements I mentioned. We are over-regulating and under-investing. So just if, in the two to three years to come, if we follow our classical agenda, we will be out of the market. I have no doubts.",

436

}

437

],

438

},

439

{

440

"role": "assistant",

441

"content": [

442

{

443

"type": "audio",

444

"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/macron.wav",

445

}

446

],

447

},

448

{

449

"role": "user",

450

"content": [{"type": "text", "text": "Hey, here is a clone from the given voice."}],

451

},

452

]

453

454

inputs = processor.apply_chat_template(

455

[conversation1, conversation2],

456

add_generation_prompt=True,

457

tokenize=True,

458

return_dict=True,

459

sampling_rate=24000,

460

return_tensors="pt",

461

).to(model.device)

462

463

outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)

464

decoded = processor.batch_decode(outputs)

465

processor.save_audio(decoded, ["output_batched_1.wav", "output_batched_2.wav"])

466

```

467

468

</details>

469

470

<details>

471

<summary>Training</summary>

472

473

By default, the model does not load the text language modeling head to save memory (~1.5GiB reduction), as it's not required for generation. When training, set `use_text_head=True` to compute loss on text tokens.

474

475

```python

476

from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration

477

478

model_id = "bosonai/higgs-audio-v2-generation-3B-base"

479

processor = AutoProcessor.from_pretrained(model_id, device_map="auto")

480

model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto", use_text_head=True)

481

482

conversation1 = [

483

{"role": "system", "content": [{"type": "text", "text": "Generate audio following instruction."}]},

484

{"role": "scene", "content": [{"type": "text", "text": "Audio is recorded from a quiet room."}]},

485

{

486

"role": "user",

487

"content": [

488

{

489

"type": "text",

490

                "text": "It was the night before my birthday. Hooray! It's almost here! It may not be a holiday, but it's the best day of the year.",

491

}

492

],

493

},

494

{

495

"role": "assistant",

496

"content": [

497

{

498

"type": "audio",

499

"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/belinda.wav",

500

}

501

],

502

},

503

]

504

505

conversation2 = [

506

{"role": "system", "content": [{"type": "text", "text": "Generate audio following instruction."}]},

507

{"role": "scene", "content": [{"type": "text", "text": "Audio is recorded from a quiet room."}]},

508

{

509

"role": "user",

510

"content": [

511

{

512

"type": "text",

513

                "text": " I would imagine so. A wand with a dragon heartstring core is capable of dazzling magic, and the bond between you and your wand should only grow stronger. Do not be surprised at your new wand's ability to perceive your intentions, particularly in a moment of need",

514

}

515

],

516

},

517

{

518

"role": "assistant",

519

"content": [

520

{

521

"type": "audio",

522

                "url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/broom_salesman.wav",

523

}

524

],

525

},

526

]

527

528

inputs = processor.apply_chat_template(

529

[conversation1, conversation2],

530

add_generation_prompt=True,

531

tokenize=True,

532

return_dict=True,

533

sampling_rate=24000,

534

return_tensors="pt",

535

output_labels=True,

536

).to(model.device)

537

538

outputs = model(**inputs)

539

outputs.loss.backward()

540

```

541

542

</details>

543

544

### Original codebase

545

546

You need to first install the [higgs-audio](https://github.com/boson-ai/higgs-audio):

547

548

```bash

549

git clone https://github.com/boson-ai/higgs-audio.git

550

551

cd higgs-audio

552

python3 -m venv higgs_audio_env

553

source higgs_audio_env/bin/activate

554

pip install -r requirements.txt

555

pip install -e .

556

```

557

558

Afterwards, try to run the following python code snippet to convert text to speech.

559

560

```python

561

from boson_multimodal.serve.serve_engine import HiggsAudioServeEngine, HiggsAudioResponse

562

from boson_multimodal.data_types import ChatMLSample, Message, AudioContent

563

564

import torch

565

import torchaudio

566

import time

567

import click

568

569

MODEL_PATH = "bosonai/higgs-audio-v2-generation-3B-base"

570

AUDIO_TOKENIZER_PATH = "bosonai/higgs-audio-v2-tokenizer"

571

572

system_prompt = (

573

    "Generate audio following instruction.\n\n<|scene_desc_start|>\nAudio is recorded from a quiet room.\n<|scene_desc_end|>"

574

)

575

576

messages = [

577

Message(

578

role="system",

579

content=system_prompt,

580

),

581

Message(

582

role="user",

583

        content="The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.",

584

),

585

]

586

device = "cuda" if torch.cuda.is_available() else "cpu"

587

588

serve_engine = HiggsAudioServeEngine(MODEL_PATH, AUDIO_TOKENIZER_PATH, device=device)

589

590

output: HiggsAudioResponse = serve_engine.generate(

591

chat_ml_sample=ChatMLSample(messages=messages),

592

max_new_tokens=1024,

593

temperature=0.3,

594

top_p=0.95,

595

top_k=50,

596

stop_strings=["<|end_of_text|>", "<|eot_id|>"],

597

)

598

torchaudio.save(f"output.wav", torch.from_numpy(output.audio)[None, :], output.sampling_rate)

599

```

600

601

You can also check https://github.com/boson-ai/higgs-audio/tree/main/examples for more example scripts.

602

603

## License

604

605

See [LICENSE](./LICENSE)

606