README.md
11.2 KB · 205 lines · markdown Raw
1 ---
2 base_model: hexgrad/Kokoro-82M
3 base_model_relation: adapter
4 pipeline_tag: text-to-speech
5 license: apache-2.0
6 language:
7 - en
8 datasets:
9 - parler-tts/libritts_r_filtered
10 - facebook/voxpopuli
11 - amphion/Emilia-Dataset
12 tags:
13 - kokoro
14 - voice-cloning
15 - adapter
16 - styletts2
17 - text-to-speech
18 ---
19
20 # kokoro-inno-clone-tuner
21
22 ![banner](https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/_u14iRNVPEU0GA3z7jk48.png)
23
24 ```python
25 pip install inno-kokoro
26 ```
27
28 Zero-shot voice tuner for [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M).
29
30 Outputs base Kokoro compatible voice packs @ `[510, 1, 256]`.
31
32
33 ## Samples
34
35 Same passage for every voice, enrolled from the references. LibriTTS-R speakers are `dev-clean` held out from training.
36
37 | voice | tuned | reference |
38 |---|---|---|
39 | am_libritts1272 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_1272_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_1272_ref.wav"></audio> |
40 | af_libritts5338 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_5338_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_5338_ref.wav"></audio> |
41 | am_libritts6241 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_6241_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_6241_ref.wav"></audio> |
42 | af_ameliaearhart | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/amelia_earhart_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/amelia_earhart_ref.wav"></audio> |
43 | bf_janegoodall | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/jane_goodall_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/jane_goodall_ref.wav"></audio> |
44 | am_vincentprice | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/vincent_price_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/vincent_price_ref.wav"></audio> |
45
46
47
48 ## Usage
49
50 Code available on Github: [remsky/inno-kokoro](https://github.com/remsky/inno-kokoro).
51
52 Integrated into [Kokoro-FastAPI](https://github.com/remsky/Kokoro-FastAPI) (v0.9.0+)
53
54 ```bash
55 pip install inno-kokoro
56 ```
57
58 Enrolling a voice:
59
60 ```python
61 from inno_kokoro.enroll import Tuner, enroll, read
62 from kokoro import KPipeline
63
64 tuner = Tuner() # pulls model.safetensors into the huggingface cache
65 pack, _ = enroll(*read("my_ref.wav"), tuner)
66
67 pipe = KPipeline(lang_code="a")
68 wav = next(pipe("Hello from a tuned voice.", voice=pack)).audio
69 ```
70
71 The pack is a plain tensor; `torch.save(pack, "voices/am_me.pt")` makes it a voice file like any other, prefixed by accent and gender like the stock packs.
72
73 The pitch-tracking ceiling is set automatically from the reference's harmonic spacing, so band-limited or archival sources land in the right octave without tuning.
74 - `enroll(..., fmax=180)` overrides it if a voice still reads the wrong register.
75
76
77 ## Architecture
78
79 Enrollment embeds an input audio sample via the baked speaker encoder and maps through `model.safetensors` to build the encoder side; prosody aspects are measured to build the predictor half. Everything loads from `model.safetensors`, no extra downloads:
80
81 - **speaker encoder**: a 6.6M-param WeSpeaker ResNet34 distilled from [UniSpeech-SAT-sv](https://huggingface.co/microsoft/unispeech-sat-base-plus-sv) embeddings
82 - matches within 0.002 speaker cosine on hold-out dev-clean (380mb fp32 -> 14mb fp16)
83 - **style head**: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
84 - **prosody blend**: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
85 - Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
86 - Packs more than 4 st from the reference's pitch are excluded.
87 - Higher graded packs are weighted as preferred during selection.
88 - **prosody head** (v0.2): a linear adjust on the blend from the reference's F0 mean and spread,
89 - fitted to deltas optimized through the frozen predictor. `enroll(..., head=False)` skips it.
90
91 Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load.
92
93 Total model size @ fp16: ~24mb
94
95 ## Limits
96
97 - English trained (LibriTTS-R) currently
98 - Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
99 - Rythym is a best-effort match to the speaker.
100 - Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
101 - Reference clip requirements:
102 - 3 second minimum, up to a max of 30s
103 - Reasonably free of audio artifacts
104 - Single speaker
105
106 ## Training and Metrics
107
108 All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted.
109
110 Trained via HF Jobs `a10g-small`; under $20 in GPU time:
111
112 <details>
113 <summary>v0.1.0 training notes</summary>
114
115 - Stage 1: mapped speaker embedding to timbre:
116 - Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding.
117 - [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
118 - Stage 2: the embedding alone missed texture and pitch; adjustments:
119 - Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt.
120 - [11k steps: dev cosine 0.934, DNSMOS 3.30]
121 - Stage 3: tuned the encoder side to match the blended predictors:
122 - Predictor half fixed to the blended rows, only the decoder half trained.
123 - [3k steps: dev cosine 0.932, DNSMOS 3.48]
124 - Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download:
125 - WeSpeaker ResNet34 plus a projection, trained on `1 - cos(student, teacher)` over LibriTTS-R crops with synthetic channel damage; the style head is untouched.
126 - [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher]
127 - Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued:
128 - Mixed real public-domain archival broadcast audio into the distillation crops.
129 - [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged]
130
131 </details>
132
133 <details>
134 <summary>v0.2.0 training notes</summary>
135
136 - Stage 1: the blend under-predicted pitch range and sentence-final falls:
137 - Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN)
138 - Targeted F0 mean, spread, IQR, pause fraction and speech time.
139 - Ridge head from reference F0 mean + spread to the fitted delta.
140 - [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st]
141 - Stage 2: the student trailed the teacher on public-speaking and in-the-wild references:
142 - Added VoxPopuli and Emilia speakers to the distillation crops.
143 - [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged]
144 - Stage 3: the timbre half had only seen LibriTTS speakers:
145 - Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw.
146 - [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged]
147
148 </details>
149
150 <br>
151
152 Measured on the shipped weights through the stock `KPipeline`: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides.
153
154 | metric | v0.1 blend only | v0.2 with head |
155 |---|---|---|
156 | speaker cosine (WavLM-sv), samples / dev-clean | 0.926 / 0.956 | 0.935 / 0.959|
157 | F0 mean, render minus reference | 0.65 st mean abs | 0.8 st mean abs |
158 | F0 spread, render minus reference | 0.65 st mean abs | 0.5 st mean abs |
159 | DNSMOS | 3.50 | 3.50 |
160
161 The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st.
162
163 ### Zero-shot identity benchmark
164
165 LibriSpeech test-clean, F5-TTS cross-sentence split (1127 utterances, 39 held-out speakers). Reference in, new sentence out, scored against the speaker's real recording.
166 - normalized: how far the render sits between sounding like a stranger (0) and a second real take of the speaker (1)
167 - RTF: RTX 4060 Ti
168
169 | system | SIM-o | normalized | ECAPA | UTMOS | RTF |
170 |---|---|---|---|---|---|
171 | ground truth (second recording) | 0.695 | 1.00 | 0.656 | 4.10 | |
172 | F5-TTS v1 base | 0.650 | 0.94 | 0.713 | 3.86 | 0.48 |
173 | StyleTTS2 (LibriTTS zero-shot) | 0.386 | 0.46 | 0.415 | 4.40 | 0.06 |
174 | **Inno v0.2** | 0.288 | 0.32 | 0.288 | 4.45 | 0.07 |
175 | OpenVoice v2 | 0.227 | 0.23 | 0.316 | 3.80 | 0.12 |
176 | Kokoro, nearest stock pack | 0.167 | 0.15 | 0.215 | 4.25 | 0.06 |
177
178 Inno gets about a third of the identity, twice as close as the nearest stock pack (with UTMOS intact and keeping Kokoro's stable natural style).
179
180 StyleTTS2 is the same architecture with an unfrozen decoder, assumed to be the ceiling of this style space.
181
182 - StyleTTS2: alpha 0.3, beta 0.7, 5 steps.
183 - OpenVoice v2: MeloTTS EN-US, tau 0.3.
184 - F5-TTS: 32 NFE, CFG 2.
185
186
187 ## Licenses and credits
188
189 Adapter weights and code: Apache-2.0, except the baked speaker encoder (`enc.*` in `model.safetensors`), which is CC BY-SA 3.0 (see below)
190 - Built on [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (Apache-2.0, StyleTTS2 architecture)
191 - The speaker encoder is a [WeSpeaker](https://github.com/wenet-e2e/wespeaker) ResNet34 (code Apache-2.0)
192 - initialised from the [VoxCeleb-trained ResNet34-LM weights](https://huggingface.co/Wespeaker/wespeaker-voxceleb-resnet34-LM) (CC BY 4.0)
193 - distilled from [microsoft/unispeech-sat-base-plus-sv](https://huggingface.co/microsoft/unispeech-sat-base-plus-sv)
194 - (Microsoft UniSpeech, [CC BY-SA 3.0 on GitHub](https://github.com/microsoft/UniSpeech/blob/main/LICENSE)); encoder weights under CC BY-SA 3.0
195 - Training data (no audio redistributed beyond the samples):
196 - [LibriTTS-R](https://www.openslr.org/141/) (CC BY 4.0), also the LibriTTS sample references
197 - [VoxPopuli](https://github.com/facebookresearch/voxpopuli) (CC0)
198 - [Emilia-YODAS](https://huggingface.co/datasets/amphion/Emilia-Dataset) EN (CC BY 4.0)
199 - Sample references:
200 - Amelia Earhart from ["Speech by Amelia Earhart"](https://www.loc.gov/item/afccal000004/), Library of Congress AFC 1986/022
201 - Jane Goodall from [BBC Woman's Hour excerpt](https://commons.wikimedia.org/wiki/File:Jane_Goodall_BBC_Radio4_Woman%27s_Hour_26_Jan_2010_p00tr6ks.flac), BBC, CC BY 3.0
202 - Vincent Price public domain from ["The House On Haunted Hill"](https://archive.org/details/The_House_On_Haunted_Hill)
203
204 Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.
205