README.md · MOSS-TTS-v1.5

1

---

2

license: apache-2.0

3

tags:

4

- text-to-speech

5

language:

6

- zh

7

- yue

8

- en

9

- ar

10

- cs

11

- da

12

- de

13

- nl

14

- es

15

- fr

16

- fi

17

- el

18

- he

19

- hi

20

- hu

21

- ja

22

- it

23

- ko

24

- mk

25

- ms

26

- ru

27

- fa

28

- pl

29

- pt

30

- sv

31

- ro

32

- sw

33

- tl

34

- th

35

- tr

36

- vi

37

---

38

# MOSS-TTS Family

39

40

41

<br>

42

43

<p align="center">

44

    

45

  <img src="https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_imgaes_demo/openmoss_x_mosi" height="50" align="middle" />

46

</p>

47

48

49

50

<div align="center">

51

  <a href="https://github.com/OpenMOSS/MOSS-TTS/tree/main"><img src="https://img.shields.io/badge/Project%20Page-GitHub-blue"></a>

52

  <a href="https://modelscope.cn/collections/OpenMOSS-Team/MOSS-TTS"><img src="https://img.shields.io/badge/ModelScope-Models-lightgrey?logo=modelscope&amp"></a>

53

  <a href="https://mosi.cn/#models"><img src="https://img.shields.io/badge/Blog-View-blue?logo=internet-explorer&amp"></a>

54

  <a href="https://arxiv.org/abs/2603.18090"><img src="https://img.shields.io/badge/Arxiv-2603.18090-red?logo=Arxiv&amp"></a>

55

56

  <a href="https://studio.mosi.cn"><img src="https://img.shields.io/badge/AIStudio-Try-green?logo=internet-explorer&amp"></a>

57

  <a href="https://studio.mosi.cn/docs/moss-tts"><img src="https://img.shields.io/badge/API-Docs-00A3FF?logo=fastapi&amp"></a>

58

<a href="https://x.com/Open_MOSS"><img src="https://img.shields.io/badge/Twitter-Follow-black?logo=x&amp"></a>

59

  <a href="https://discord.gg/fvm5TaWjU3"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&amp"></a>

60

</div>

61

62

63

# MOSS-TTS-v1.5

64

65

**MOSS-TTS-v1.5** is continued from [MOSS-TTS 1.0](https://huggingface.co/OpenMOSS-Team/MOSS-TTS). It preserves the main 1.0 capabilities, including zero-shot voice cloning, long-form speech generation, token-level duration control, Pinyin/IPA pronunciation control, multilingual synthesis, and code-switching. For the full 1.0 feature walkthrough, input schema, decoding hyperparameters, and evaluation tables, please refer to the [MOSS-TTS 1.0 README](https://huggingface.co/OpenMOSS-Team/MOSS-TTS).

66

67

Compared with MOSS-TTS 1.0, v1.5 focuses on the following improvements:

68

69

- **Stronger multilingual synthesis with language tags**: when the `language` field is omitted, v1.5 may improve some languages and regress slightly on others compared with 1.0. When the language is specified, v1.5 is stronger than 1.0 on almost all supported languages. Set the tag when building the user message, for example `processor.build_user_message(text=text_fr, language="French")`.

70

- **More stable voice cloning**: v1.5 improves speaker similarity and reduces cloning variance, making repeated generations more consistent.

71

- **Better long-reference, short-text cloning**: v1.5 handles scenarios where the reference audio is much longer than the target text more reliably than 1.0.

72

- **More stable punctuation-following prosody**: v1.5 follows punctuation-driven pauses more closely, especially in long sentences.

73

- **Explicit pause control**: v1.5 supports inline pause markers such as `"[pause 3.2s]"`. For example, `我今天学习了一首中国的古诗，它的名字是[pause 3.2s]静夜思！` inserts an explicit 3.2s pause before `静夜思`.

74

75

## Supported Languages

76

77

MOSS-TTS-v1.5 currently supports **31 languages**. It keeps the 20 languages supported by [MOSS-TTS 1.0](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) and extends multilingual continued training to additional languages including Cantonese, Dutch, Finnish, Hindi, Macedonian, Malay, Romanian, Swahili, Tagalog, Thai, and Vietnamese.

78

79

80

|---|---|---|---|---|---|---|---|---|

81

| Chinese | zh | 🇨🇳 | Cantonese | yue | 🇭🇰 | English | en | 🇺🇸 |

82

| Arabic | ar | 🇸🇦 | Czech | cs | 🇨🇿 | Danish | da | 🇩🇰 |

83

| Dutch | nl | 🇳🇱 | Finnish | fi | 🇫🇮 | French | fr | 🇫🇷 |

84

| German | de | 🇩🇪 | Greek | el | 🇬🇷 | Hebrew | he | 🇮🇱 |

85

| Hindi | hi | 🇮🇳 | Hungarian | hu | 🇭🇺 | Italian | it | 🇮🇹 |

86

| Japanese | ja | 🇯🇵 | Korean | ko | 🇰🇷 | Macedonian | mk | 🇲🇰 |

87

| Malay | ms | 🇲🇾 | Persian (Farsi) | fa | 🇮🇷 | Polish | pl | 🇵🇱 |

88

| Portuguese | pt | 🇵🇹 | Romanian | ro | 🇷🇴 | Russian | ru | 🇷🇺 |

89

| Spanish | es | 🇪🇸 | Swahili | sw | 🇹🇿 | Swedish | sv | 🇸🇪 |

90

| Tagalog | tl | 🇵🇭 | Thai | th | 🇹🇭 | Turkish | tr | 🇹🇷 |

91

| Vietnamese | vi | 🇻🇳 | | | | | | |

92

93

94

## Quick Start

95

96

### Environment Setup

97

98

We recommend a clean, isolated Python environment with **Transformers 5.0.0** to avoid dependency conflicts.

99

100

```bash

101

conda create -n moss-tts python=3.12 -y

102

conda activate moss-tts

103

```

104

105

Install all required dependencies:

106

107

```bash

108

git clone https://github.com/OpenMOSS/MOSS-TTS.git

109

cd MOSS-TTS

110

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e .

111

```

112

113

#### (Optional) Install FlashAttention 2

114

115

For better speed and lower GPU memory usage, you can install FlashAttention 2 if your hardware supports it.

116

117

```bash

118

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]"

119

```

120

121

If your machine has limited RAM and many CPU cores, you can cap build parallelism:

122

123

```bash

124

MAX_JOBS=4 pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]"

125

```

126

127

Notes:

128

- Dependencies are managed in `pyproject.toml`, which currently pins `torch==2.9.1+cu128` and `torchaudio==2.9.1+cu128`.

129

- If FlashAttention 2 fails to build on your machine, you can skip it and use the default attention backend.

130

- FlashAttention 2 is only available on supported GPUs and is typically used with `torch.float16` or `torch.bfloat16`.

131

132

133

### Basic Usage

134

135

> Tip: MOSS-TTS-v1.5 uses the same generation API as the 1.0 **MossTTSDelay-8B** checkpoint. For multilingual inputs, set `language` whenever the language is known.

136

137

MOSS-TTS provides a convenient `generate` interface for rapid usage. The examples below cover:

138

1. Direct generation (Chinese / English / multilingual text with language tags / Pinyin / IPA)

139

2. Voice cloning

140

3. Duration control

141

4. Explicit pause control with `[pause X.Ys]`

142

143

```python

144

from pathlib import Path

145

import importlib.util

146

import torch

147

import torchaudio

148

from transformers import AutoModel, AutoProcessor

149

# Disable the broken cuDNN SDPA backend

150

torch.backends.cuda.enable_cudnn_sdp(False)

151

# Keep these enabled as fallbacks

152

torch.backends.cuda.enable_flash_sdp(True)

153

torch.backends.cuda.enable_mem_efficient_sdp(True)

154

torch.backends.cuda.enable_math_sdp(True)

155

156

157

pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS-v1.5"

158

device = "cuda" if torch.cuda.is_available() else "cpu"

159

dtype = torch.bfloat16 if device == "cuda" else torch.float32

160

161

def resolve_attn_implementation() -> str:

162

# Prefer FlashAttention 2 when package + device conditions are met.

163

if (

164

device == "cuda"

165

and importlib.util.find_spec("flash_attn") is not None

166

and dtype in {torch.float16, torch.bfloat16}

167

):

168

major, _ = torch.cuda.get_device_capability()

169

if major >= 8:

170

return "flash_attention_2"

171

172

# CUDA fallback: use PyTorch SDPA kernels.

173

if device == "cuda":

174

return "sdpa"

175

176

# CPU fallback.

177

return "eager"

178

179

180

attn_implementation = resolve_attn_implementation()

181

print(f"[INFO] Using attn_implementation={attn_implementation}")

182

183

processor = AutoProcessor.from_pretrained(

184

pretrained_model_name_or_path,

185

trust_remote_code=True,

186

)

187

processor.audio_tokenizer = processor.audio_tokenizer.to(device)

188

189

text_1 = "亲爱的你，\n你好呀。\n\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。\n\n首先，我想祝你——\n每天都能平平安安、快快乐乐。\n\n希望你早上醒来的时候，\n窗外有光，屋子里很安静，\n你的心是轻轻的，没有着急，也没有害怕。\n\n希望你吃饭的时候胃口很好，\n走路的时候脚步稳稳，\n晚上睡觉的时候，能做一个又一个甜甜的梦。\n\n我希望你能一直保持好奇心。\n对世界充满问题，\n对天空、星星、花草、书本和故事感兴趣。\n当你问“为什么”的时候，\n希望总有人愿意认真地听你说话。\n\n我也希望你学会温柔。\n温柔地对待朋友，\n温柔地对待小动物，\n也温柔地对待自己。\n\n如果有一天你犯了错，\n请不要太快责怪自己，\n因为每一个认真成长的人，\n都会在路上慢慢学会更好的方法。\n\n愿你拥有勇气。\n当你站在陌生的地方时，\n当你第一次举手发言时，\n当你遇到困难、感到害怕的时候，\n希望你能轻轻地告诉自己：\n“我可以试一试。”\n\n就算没有一次成功，也没有关系。\n失败不是坏事，\n它只是告诉你，你正在努力。\n\n我希望你学会分享快乐。\n把开心的事情告诉别人，\n把笑声送给身边的人，\n因为快乐被分享的时候，\n会变得更大、更亮。\n\n如果有一天你感到难过，\n我希望你知道——\n难过并不丢脸，\n哭泣也不是软弱。\n\n愿你能找到一个安全的地方，\n慢慢把心里的话说出来，\n然后再一次抬起头，看见希望。\n\n我还希望你能拥有梦想。\n这个梦想也许很大，\n也许很小，\n也许现在还说不清楚。\n\n没关系。\n梦想会和你一起长大，\n在时间里慢慢变得清楚。\n\n最后，我想送你一个最最重要的祝福：\n\n愿你被世界温柔对待，\n也愿你成为一个温柔的人。\n\n愿你的每一天，\n都值得被记住，\n都值得被珍惜。\n\n亲爱的你，\n请记住，\n你是独一无二的，\n你已经很棒了，\n而你的未来，\n一定会慢慢变得闪闪发光。\n\n祝你健康、勇敢、幸福，\n祝你永远带着笑容向前走。"

190

text_2 = "We stand on the threshold of the AI era.\nArtificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."

191

text_3 = "nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？"

192

text_4 = "nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？"

193

text_5 = "您好，请问您来自哪 zuo4 cheng2 shi4？"

194

text_6 = "/həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/"

195

text_7 = "Bonjour, je voudrais essayer une voix française naturelle et stable."

196

text_8 = "我今天学习了一首中国的古诗，它的名字是[pause 3.2s]静夜思！"

197

198

# Use audio from ./assets/audio to avoid downloading from the cloud.

199

ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"

200

ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"

201

202

conversations = [

203

# Direct TTS (no reference). Language tags are recommended in v1.5.

204

[processor.build_user_message(text=text_1)],

205

[processor.build_user_message(text=text_2)],

206

# Direct TTS (no reference). For languages other than Chinese and English, it is recommended to use language tags.

207

[processor.build_user_message(text=text_7, language="French")],

208

# Pinyin or IPA input

209

[processor.build_user_message(text=text_3)],

210

[processor.build_user_message(text=text_4)],

211

[processor.build_user_message(text=text_5)],

212

[processor.build_user_message(text=text_6)],

213

# Explicit pause control. Use [pause X.Ys], such as [pause 3.2s].

214

[processor.build_user_message(text=text_8)],

215

# Voice cloning (with reference)

216

[processor.build_user_message(text=text_1, reference=[ref_audio_1])],

217

[processor.build_user_message(text=text_2, reference=[ref_audio_2])],

218

# Duration control

219

[processor.build_user_message(text=text_2, tokens=325)],

220

[processor.build_user_message(text=text_2, tokens=600)],

221

]

222

223

model = AutoModel.from_pretrained(

224

pretrained_model_name_or_path,

225

trust_remote_code=True,

226

# If FlashAttention 2 is installed, you can set attn_implementation="flash_attention_2"

227

attn_implementation=attn_implementation,

228

torch_dtype=dtype,

229

).to(device)

230

model.eval()

231

232

batch_size = 1

233

234

save_dir = Path("inference_root")

235

save_dir.mkdir(exist_ok=True, parents=True)

236

sample_idx = 0

237

with torch.no_grad():

238

for start in range(0, len(conversations), batch_size):

239

batch_conversations = conversations[start : start + batch_size]

240

batch = processor(batch_conversations, mode="generation")

241

input_ids = batch["input_ids"].to(device)

242

attention_mask = batch["attention_mask"].to(device)

243

244

outputs = model.generate(

245

input_ids=input_ids,

246

attention_mask=attention_mask,

247

max_new_tokens=4096,

248

)

249

250

for message in processor.decode(outputs):

251

audio = message.audio_codes_list[0]

252

out_path = save_dir / f"sample{sample_idx}.wav"

253

sample_idx += 1

254

torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)

255

256

```

257

258

## More Usage

259

260

MOSS-TTS-v1.5 is API-compatible with MOSS-TTS 1.0. For continuation with prefix audio, detailed `UserMessage` and `AssistantMessage` fields, generation hyperparameters, Pinyin/IPA preprocessing examples, and evaluation results, see the [MOSS-TTS 1.0 README](https://huggingface.co/OpenMOSS-Team/MOSS-TTS).

261