README.md
12.7 KB · 391 lines · markdown Raw
1 ---
2 language:
3 - en
4 library_name: nemo
5 datasets:
6 - librispeech_asr
7 - fisher_corpus
8 - Switchboard-1
9 - WSJ-0
10 - WSJ-1
11 - National-Singapore-Corpus-Part-1
12 - National-Singapore-Corpus-Part-6
13 - vctk
14 - voxpopuli
15 - europarl
16 - multilingual_librispeech
17 - mozilla-foundation/common_voice_8_0
18 - MLCommons/peoples_speech
19 thumbnail: null
20 tags:
21 - transformers
22 - automatic-speech-recognition
23 - speech
24 - audio
25 - FastConformer
26 - Conformer
27 - pytorch
28 - NeMo
29 - hf-asr-leaderboard
30 - ctc
31 license: cc-by-4.0
32 widget:
33 - example_title: Librispeech sample 1
34 src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
35 - example_title: Librispeech sample 2
36 src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
37 model-index:
38 - name: parakeet-ctc-1.1b
39 results:
40 - task:
41 name: Automatic Speech Recognition
42 type: automatic-speech-recognition
43 dataset:
44 name: AMI (Meetings test)
45 type: edinburghcstr/ami
46 config: ihm
47 split: test
48 args:
49 language: en
50 metrics:
51 - name: Test WER
52 type: wer
53 value: 15.62
54 - task:
55 name: Automatic Speech Recognition
56 type: automatic-speech-recognition
57 dataset:
58 name: Earnings-22
59 type: revdotcom/earnings22
60 split: test
61 args:
62 language: en
63 metrics:
64 - name: Test WER
65 type: wer
66 value: 13.69
67 - task:
68 name: Automatic Speech Recognition
69 type: automatic-speech-recognition
70 dataset:
71 name: GigaSpeech
72 type: speechcolab/gigaspeech
73 split: test
74 args:
75 language: en
76 metrics:
77 - name: Test WER
78 type: wer
79 value: 10.27
80 - task:
81 name: Automatic Speech Recognition
82 type: automatic-speech-recognition
83 dataset:
84 name: LibriSpeech (clean)
85 type: librispeech_asr
86 config: other
87 split: test
88 args:
89 language: en
90 metrics:
91 - name: Test WER
92 type: wer
93 value: 1.83
94 - task:
95 name: Automatic Speech Recognition
96 type: automatic-speech-recognition
97 dataset:
98 name: LibriSpeech (other)
99 type: librispeech_asr
100 config: other
101 split: test
102 args:
103 language: en
104 metrics:
105 - name: Test WER
106 type: wer
107 value: 3.54
108 - task:
109 type: Automatic Speech Recognition
110 name: automatic-speech-recognition
111 dataset:
112 name: SPGI Speech
113 type: kensho/spgispeech
114 config: test
115 split: test
116 args:
117 language: en
118 metrics:
119 - name: Test WER
120 type: wer
121 value: 4.2
122 - task:
123 type: Automatic Speech Recognition
124 name: automatic-speech-recognition
125 dataset:
126 name: tedlium-v3
127 type: LIUM/tedlium
128 config: release1
129 split: test
130 args:
131 language: en
132 metrics:
133 - name: Test WER
134 type: wer
135 value: 3.54
136 - task:
137 name: Automatic Speech Recognition
138 type: automatic-speech-recognition
139 dataset:
140 name: Vox Populi
141 type: facebook/voxpopuli
142 config: en
143 split: test
144 args:
145 language: en
146 metrics:
147 - name: Test WER
148 type: wer
149 value: 6.53
150 - task:
151 type: Automatic Speech Recognition
152 name: automatic-speech-recognition
153 dataset:
154 name: Mozilla Common Voice 9.0
155 type: mozilla-foundation/common_voice_9_0
156 config: en
157 split: test
158 args:
159 language: en
160 metrics:
161 - name: Test WER
162 type: wer
163 value: 9.02
164 metrics:
165 - wer
166 pipeline_tag: automatic-speech-recognition
167 ---
168
169 # Parakeet CTC 1.1B (en)
170
171 <style>
172 img {
173 display: inline;
174 }
175 </style>
176
177 [![Model architecture](https://img.shields.io/badge/Model_Arch-FastConformer--CTC-lightgrey#model-badge)](#model-architecture)
178 | [![Model size](https://img.shields.io/badge/Params-1.1B-lightgrey#model-badge)](#model-architecture)
179 | [![Language](https://img.shields.io/badge/Language-en-lightgrey#model-badge)](#datasets)
180
181
182 `parakeet-ctc-1.1b` is an ASR model that transcribes speech in lower case English alphabet. This model is jointly developed by [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) and [Suno.ai](https://www.suno.ai/) teams.
183 It is an XXL version of FastConformer CTC [1] (around 1.1B parameters) model.
184 See the [model architecture](#model-architecture) section and [NeMo documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/models.html#fast-conformer) for complete architecture details.
185
186 ## NVIDIA NeMo: Training
187
188 To train, fine-tune or play with the model you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). We recommend you install it after you've installed latest PyTorch version.
189 ```
190 pip install nemo_toolkit['all']
191 ```
192
193 ## How to Use this Model
194
195 There are several ways to use this model. Choose the one that fits your needs.
196
197 ### Run locally with NeMo-Speech.cpp
198
199 [NeMo-Speech.cpp](https://github.com/NVIDIA/NeMo-Speech.cpp) provides a
200 lightweight native C++ runtime for local inference with
201 this model. After [installing the runtime](https://github.com/NVIDIA/NeMo-Speech.cpp#installation):
202
203 ```bash
204 hf download nvidia/parakeet-ctc-1.1b \
205 parakeet-ctc-1.1b.q8_0.gguf \
206 --local-dir models
207
208 nemo-speech transcribe audio.wav \
209 --model models/parakeet-ctc-1.1b.q8_0.gguf
210 ```
211
212 See the [NeMo-Speech.cpp documentation](https://github.com/NVIDIA/NeMo-Speech.cpp)
213 for more details.
214
215 ### NVIDIA NeMo
216
217 The model is available for use in the NeMo toolkit [3], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset. Moreover, you can now run Parakeet CTC natively with [Transformers](https://github.com/huggingface/transformers) 🤗.
218
219 #### Automatically instantiate the model
220
221 ```python
222 import nemo.collections.asr as nemo_asr
223 asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name="nvidia/parakeet-ctc-1.1b")
224 ```
225
226 #### Transcribing using NeMo
227 First, let's get a sample
228 ```
229 wget https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav
230 ```
231 Then simply do:
232 ```
233 asr_model.transcribe(['2086-149220-0033.wav'])
234 ```
235
236 ### Transcribing using [Transformers](https://github.com/huggingface/transformers) 🤗
237
238 Make sure to install `transformers` from source.
239
240 ```bash
241 pip install git+https://github.com/huggingface/transformers
242 ```
243
244 <details>
245 <summary>➡️ Pipeline usage</summary>
246
247 ```python
248 from transformers import pipeline
249
250 pipe = pipeline("automatic-speech-recognition", model="nvidia/parakeet-ctc-1.1b")
251 out = pipe("https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3")
252 print(out)
253 ```
254 </details>
255
256 <details>
257 <summary>➡️ AutoModel</summary>
258
259 ```python
260 from transformers import AutoModelForCTC, AutoProcessor
261 from datasets import load_dataset, Audio
262 import torch
263
264 device = "cuda" if torch.cuda.is_available() else "cpu"
265
266 processor = AutoProcessor.from_pretrained("nvidia/parakeet-ctc-1.1b")
267 model = AutoModelForCTC.from_pretrained("nvidia/parakeet-ctc-1.1b", dtype="auto", device_map=device)
268
269 ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
270 ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
271 speech_samples = [el['array'] for el in ds["audio"][:5]]
272
273 inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
274 inputs.to(model.device, dtype=model.dtype)
275 outputs = model.generate(**inputs)
276 print(processor.batch_decode(outputs))
277 ```
278 </details>
279
280 <details>
281 <summary>➡️ Training</summary>
282
283 ```python
284 from transformers import AutoModelForCTC, AutoProcessor
285 from datasets import load_dataset, Audio
286 import torch
287
288 device = "cuda" if torch.cuda.is_available() else "cpu"
289
290 processor = AutoProcessor.from_pretrained("nvidia/parakeet-ctc-1.1b")
291 model = AutoModelForCTC.from_pretrained("nvidia/parakeet-ctc-1.1b", dtype="auto", device_map=device)
292
293 ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
294 ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
295 speech_samples = [el['array'] for el in ds["audio"][:5]]
296 text_samples = [el for el in ds["text"][:5]]
297
298 # passing `text` to the processor will prepare inputs' `labels` key
299 inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
300 inputs.to(device, dtype=model.dtype)
301
302 outputs = model(**inputs)
303 outputs.loss.backward()
304 ```
305 </details>
306
307 For more details about usage, the refer to [Transformers' documentation](https://huggingface.co/docs/transformers/en/index).
308
309 ### Transcribing many audio files
310
311 ```shell
312 python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py
313 pretrained_name="nvidia/parakeet-ctc-1.1b"
314 audio_dir="<DIRECTORY CONTAINING AUDIO FILES>"
315 ```
316
317 ### Input
318
319 This model accepts 16000 Hz mono-channel audio (wav files) as input.
320
321 ### Output
322
323 This model provides transcribed speech as a string for a given audio sample.
324
325 ## Model Architecture
326
327 FastConformer [1] is an optimized version of the Conformer model with 8x depthwise-separable convolutional downsampling. The model is trained using CTC loss. You may find more information on the details of FastConformer here: [Fast-Conformer Model](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/models.html#fast-conformer).
328
329 ## Training
330
331 The NeMo toolkit [3] was used for training the models for over several hundred epochs. These model are trained with this [example script](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py) and this [base config](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/fastconformer/fast-conformer_ctc_bpe.yaml).
332
333 The tokenizers for these models were built using the text transcripts of the train set with this [script](https://github.com/NVIDIA/NeMo/blob/main/scripts/tokenizers/process_asr_text_tokenizer.py).
334
335 ### Datasets
336
337 The model was trained on 64K hours of English speech collected and prepared by NVIDIA NeMo and Suno teams.
338
339 The training dataset consists of private subset with 40K hours of English speech plus 24K hours from the following public datasets:
340
341 - Librispeech 960 hours of English speech
342 - Fisher Corpus
343 - Switchboard-1 Dataset
344 - WSJ-0 and WSJ-1
345 - National Speech Corpus (Part 1, Part 6)
346 - VCTK
347 - VoxPopuli (EN)
348 - Europarl-ASR (EN)
349 - Multilingual Librispeech (MLS EN) - 2,000 hour subset
350 - Mozilla Common Voice (v7.0)
351 - People's Speech - 12,000 hour subset
352
353 ## Performance
354
355 The performance of Automatic Speech Recognition models is measuring using Word Error Rate. Since this dataset is trained on multiple domains and a much larger corpus, it will generally perform better at transcribing audio in general.
356
357 The following tables summarizes the performance of the available models in this collection with the CTC decoder. Performances of the ASR models are reported in terms of Word Error Rate (WER%) with greedy decoding.
358
359 |**Version**|**Tokenizer**|**Vocabulary Size**|**AMI**|**Earnings-22**|**Giga Speech**|**LS test-clean**|**SPGI Speech**|**TEDLIUM-v3**|**Vox Populi**|**Common Voice**|
360 |---------|-----------------------|-----------------|---------------|---------------|------------|-----------|-----|-------|------|------|
361 | 1.22.0 | SentencePiece Unigram | 1024 | 15.62 | 13.69 | 10.27 | 1.83 | 3.54 | 4.20 | 3.54 | 6.53 | 9.02 |
362
363 These are greedy WER numbers without external LM. More details on evaluation can be found at [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
364
365 ## NVIDIA Riva: Deployment
366
367 [NVIDIA Riva](https://developer.nvidia.com/riva), is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded.
368 Additionally, Riva provides:
369
370 * World-class out-of-the-box accuracy for the most common languages with model checkpoints trained on proprietary data with hundreds of thousands of GPU-compute hours
371 * Best in class accuracy with run-time word boosting (e.g., brand and product names) and customization of acoustic model, language model, and inverse text normalization
372 * Streaming speech recognition, Kubernetes compatible scaling, and enterprise-grade support
373
374 Although this model isn’t supported yet by Riva, the [list of supported models is here](https://huggingface.co/models?other=Riva).
375 Check out [Riva live demo](https://developer.nvidia.com/riva#demos).
376
377 ## References
378 [1] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
379
380 [2] [Google Sentencepiece Tokenizer](https://github.com/google/sentencepiece)
381
382 [3] [NVIDIA NeMo Toolkit](https://github.com/NVIDIA/NeMo)
383
384 [4] [Suno.ai](https://suno.ai/)
385
386 [5] [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
387
388
389 ## Licence
390
391 License to use this model is covered by the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). By downloading the public and release version of the model, you accept the terms and conditions of the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) license.