AGENTS.md · higgs-tts-3-4b

1

# AGENTS.md — Higgs Audio v3 TTS (4B)

2

3

> Operational guide for AI coding agents. This file is **self-contained**: you can act on it

4

> even before cloning this repo. For model background, benchmarks, the full language list, and

5

> citation, see the **[model card README](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b/blob/main/README.md)** — don't duplicate that narrative here.

6

7

Higgs Audio v3 TTS is a **4B-parameter, conversational text-to-speech model**: expressive,

8

low-latency, 100+ languages, zero-shot voice cloning, and inline control over emotion / prosody /

9

pauses / sound effects mid-utterance.

10

11

---

12

13

## Step 0 — Pick the right path (read this first)

14

15

Choose by constraint, not by habit:

16

17

| Goal | Use | Entry point |

18

|------|-----|-------------|

19

| Just hear it / try preset voices & avatars | **Live Demo** | https://boson.ai/workspace/avatar |

20

| Integrate quickly, no GPU, your own voice | **Hosted API** | https://docs.boson.ai/models/higgs-audio-tts/overview |

21

| Data privacy, custom testing, full control (NVIDIA GPU) | **Self-host (SGLang-Omni)** | https://lmsys.org/blog/2026-06-04-higgs-audio-v3-tts/ |

22

| Run locally on a Mac (Apple Silicon, no NVIDIA GPU) | **Self-host (MLX-Audio)** | https://github.com/Blaizzy/mlx-audio |

23

| Node-based UI / visual workflow | **ComfyUI (community)** | https://github.com/Saganaki22/Higgs_v3-TTS-ComfyUI |

24

| Inspect weights / config / tokenizer | **Model card (this repo)** | https://huggingface.co/bosonai/higgs-audio-v3-tts-4b |

25

26

Deep dive on everything: **Technical blog** → https://boson.ai/blog/higgs-audio-v3-tts

27

28

---

29

30

## Path A — Hosted API (fastest, no GPU)

31

32

> **Authoritative docs:** https://docs.boson.ai/models/higgs-audio-tts/overview

33

> Get an API key, full field reference, and Python/TypeScript SDK examples there.

34

> An agent cannot invent a key — if `BOSON_API_KEY` is unset, stop and point the user to this page.

35

36

```bash

37

export BOSON_API_KEY=bai-xxxx # obtain from https://docs.boson.ai (key format: bai-...)

38

```

39

40

Basic synthesis:

41

42

```bash

43

curl https://api.boson.ai/v1/audio/speech \

44

-H "Authorization: Bearer $BOSON_API_KEY" \

45

-H "Content-Type: application/json" \

46

-d '{"model":"higgs-audio-v3-tts","input":"Hello, this is a test."}' \

47

--output out.mp3

48

```

49

50

Request fields:

51

52

| Field | Notes |

53

|-------|-------|

54

| `model` | `"higgs-audio-v3-tts"` |

55

| `input` | text to synthesize (**required**) |

56

| `voice` | preset speaker, e.g. `"jake"` |

57

| `ref_audio` + `ref_text` | URL/base64 clip + its transcript → **voice cloning** |

58

| `response_format` | `"mp3"` (default) or `"pcm"` (use `pcm` for low-latency streaming) |

59

| `stream` | `true` for SSE streaming |

60

61

> Verify exact field names/limits against the API docs before shipping — the hosted API evolves

62

> independently of these weights.

63

64

---

65

66

## Path B — Self-host with SGLang-Omni

67

68

### B0 — Preflight: confirm hardware first (do this before pulling anything)

69

70

Performance numbers are benchmarked on **1× H100 (80 GB)**. The model is also **confirmed to run on

71

1× A100 40 GB** — so **~40 GB VRAM is a known-good floor**. Smaller GPUs are **untested** (no data,

72

not "won't work"). Before deploying:

73

74

```bash

75

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv # GPU present? how much VRAM?

76

docker --version && docker info | grep -i runtime # Docker + NVIDIA runtime ready?

77

df -h . # disk for the ~4B weights + image

78

```

79

80

Rules for the agent:

81

- **No NVIDIA GPU** → stop this path. On an **Apple Silicon Mac**, use **Path C (MLX-Audio)**;

82

for a node-based UI, see **Path D (ComfyUI)**; otherwise use **Path A (hosted API)**.

83

- **≥ 40 GB VRAM (e.g. A100 40 GB, H100)** → known-good; proceed.

84

- **24 GB (e.g. RTX 4090)** → *reported* to work, **not officially verified**. The ~4B weights fit,

85

but expect to lower concurrency / `max_new_tokens` and watch for OOM at the `serve` step.

86

- **< 24 GB VRAM** → untested. It *may* still run (4B model), but no one has verified it. Warn the

87

user, and be ready to lower concurrency / `max_new_tokens` if you hit OOM at the `serve` step.

88

- **Don't assume** a VRAM number — confirm against the SGLang-Omni cookbook / blog before promising

89

a given GPU will work: https://lmsys.org/blog/2026-06-04-higgs-audio-v3-tts/

90

91

### B1 — Install & serve

92

93

```bash

94

# 1. Container

95

docker pull lmsysorg/sglang-omni:dev

96

docker run -it --gpus all --shm-size 32g --ipc host --network host --privileged \

97

lmsysorg/sglang-omni:dev /bin/zsh

98

99

# 2. Engine

100

git clone git@github.com:sgl-project/sglang-omni.git && cd sglang-omni

101

uv venv .venv -p 3.12 && source .venv/bin/activate

102

uv pip install -v -e .

103

104

# 3. Weights

105

hf download bosonai/higgs-audio-v3-tts-4b

106

107

# 4. Serve (OpenAI-compatible audio endpoint)

108

sgl-omni serve --model-path bosonai/higgs-audio-v3-tts-4b --port 8000

109

```

110

111

Call the local server:

112

113

```bash

114

curl -X POST http://localhost:8000/v1/audio/speech \

115

-H "Content-Type: application/json" \

116

-d '{"input": "Hello, how are you?"}' \

117

--output output.wav

118

```

119

120

**Recommended sampling (voice cloning):** `temperature: 0.8`, `top_k: 50`, `max_new_tokens: 1024`.

121

122

Cookbook reference: https://sgl-project.github.io/sglang-omni/cookbook/higgs_tts.html

123

124

---

125

126

## Path C — Apple Silicon Mac via MLX-Audio (no NVIDIA GPU)

127

128

For Macs there is **no CUDA / Docker path** — use **MLX-Audio**, an Apple-MLX-native TTS library

129

that runs the model directly on M-series GPUs: https://github.com/Blaizzy/mlx-audio

130

131

**Hardware (first-hand, measured):** confirmed on an **M1 / 32 GB**, with a peak memory footprint of

132

only **~9–12 GB** — comfortably within reach of typical Apple Silicon laptops, no discrete GPU needed.

133

134

```bash

135

pip install mlx-audio # requires Apple Silicon (M1/M2/M3/M4) + macOS

136

```

137

138

Drive the model through MLX-Audio's CLI / Python API per its README — see

139

https://github.com/Blaizzy/mlx-audio for the exact `generate` command and supported flags.

140

141

> Mac-only. On Linux/NVIDIA use **Path B**; with no local accelerator at all, use **Path A**.

142

143

---

144

145

## Path D — ComfyUI node-based UI (community)

146

147

A community integration exposes the model as ComfyUI nodes (text-to-speech in a visual,

148

node-based workflow), with a drag-and-drop workflow file for immediate use:

149

150

- **Repo:** https://github.com/Saganaki22/Higgs_v3-TTS-ComfyUI (by Saganaki22)

151

152

> **Third-party, not maintained by Boson.** Follow that repo's README for install/usage, and verify

153

> it against the version of the weights you intend to run. Surfaced in the model's HF discussions:

154

> https://huggingface.co/bosonai/higgs-audio-v3-tts-4b/discussions/4

155

156

---

157

158

## Control tags — how to write target text

159

160

Embed tags directly in the `input` text to steer emotion, prosody, style, and sound effects.

161

Format is always `<|category:tag|>`, with two placements:

162

163

- **Sentence-level** (emotion / style / prosody speed·pitch·expressive) → put at the sentence start.

164

- **Inline** (sfx, and prosody `pause` / `long_pause`) → insert at the exact spot in the sentence.

165

- **`sfx` gotcha:** `<|sfx:cough|>Ahem, ...` — tag first, onomatopoeia attached, **no space**.

166

167

```

168

<|emotion:elation|>Welcome aboard, we are thrilled to have you here!

169

<|emotion:elation|><|sfx:laughter|>Haha, welcome, we're so happy you're here!

170

Hello there <|prosody:pause|> and welcome to the show.

171

```

172

173

> **Full 43-tag catalog + rules + examples → [PROMPTING.md](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b/blob/main/PROMPTING.md).**

174

> Only recognized tags work — anything else degrades output or gets read literally.

175

176

For chat formatting, use **[`chat_template.jinja`](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b/blob/main/chat_template.jinja)** from the model repo (and the API docs);

177

**do not hand-assemble the chat prompt** — go through the template.

178

179

## Language codes

180

181

Only the ISO codes listed in README's supported-languages section are reliable. Codes outside that

182

list fall back / degrade. → see the [model card README](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b/blob/main/README.md) (`## Supported Languages`).

183

184

---

185

186

## Repo contents (what's actually here)

187

188

This repo (`https://huggingface.co/bosonai/higgs-audio-v3-tts-4b`) is **weights + config**, not an

189

inference codebase:

190

191

- `config.json`, `model.safetensors(.index.json)` — model weights & shape

192

- `chat_template.jinja` — **authoritative** prompt/chat formatting; respect it

193

- `tokenizer.json`, `tokenizer_config.json` — tokenizer

194

- `README.md` — HuggingFace model card (capabilities, benchmarks, languages, citation)

195

- `LICENSE` — see red line below

196

197

## Do / Don't

198

199

- ✅ Use `chat_template.jinja` for prompt construction; use the OpenAI-compatible `/v1/audio/speech` shape.

200

- ✅ Use `pcm` + `stream` for real-time / conversational latency.

201

- ❌ **Don't use commercially.** License is research & non-commercial

202

  (`boson-higgs-audio-v3-research-and-non-commercial-license`) — see [LICENSE](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b/blob/main/LICENSE).

203

- ❌ Don't hardcode the 100-language claim as "any code works" — validate against the supported list.

204

205

## Pointers (don't duplicate — link)

206

207

All on the model card: `https://huggingface.co/bosonai/higgs-audio-v3-tts-4b/blob/main/README.md`

208

209

- Benchmarks / WER-CER tables → README `## Evaluation Benchmarks`

210

- Full language list → README `## Supported Languages`

211

- Control-token catalog → README `## Control Tokens`

212

- Citation → README `## Citation`

213