README.md
| 1 | --- |
| 2 | tags: |
| 3 | - mms |
| 4 | language: |
| 5 | - ab |
| 6 | - af |
| 7 | - ak |
| 8 | - am |
| 9 | - ar |
| 10 | - as |
| 11 | - av |
| 12 | - ay |
| 13 | - az |
| 14 | - ba |
| 15 | - bm |
| 16 | - be |
| 17 | - bn |
| 18 | - bi |
| 19 | - bo |
| 20 | - sh |
| 21 | - br |
| 22 | - bg |
| 23 | - ca |
| 24 | - cs |
| 25 | - ce |
| 26 | - cv |
| 27 | - ku |
| 28 | - cy |
| 29 | - da |
| 30 | - de |
| 31 | - dv |
| 32 | - dz |
| 33 | - el |
| 34 | - en |
| 35 | - eo |
| 36 | - et |
| 37 | - eu |
| 38 | - ee |
| 39 | - fo |
| 40 | - fa |
| 41 | - fj |
| 42 | - fi |
| 43 | - fr |
| 44 | - fy |
| 45 | - ff |
| 46 | - ga |
| 47 | - gl |
| 48 | - gn |
| 49 | - gu |
| 50 | - zh |
| 51 | - ht |
| 52 | - ha |
| 53 | - he |
| 54 | - hi |
| 55 | - sh |
| 56 | - hu |
| 57 | - hy |
| 58 | - ig |
| 59 | - ia |
| 60 | - ms |
| 61 | - is |
| 62 | - it |
| 63 | - jv |
| 64 | - ja |
| 65 | - kn |
| 66 | - ka |
| 67 | - kk |
| 68 | - kr |
| 69 | - km |
| 70 | - ki |
| 71 | - rw |
| 72 | - ky |
| 73 | - ko |
| 74 | - kv |
| 75 | - lo |
| 76 | - la |
| 77 | - lv |
| 78 | - ln |
| 79 | - lt |
| 80 | - lb |
| 81 | - lg |
| 82 | - mh |
| 83 | - ml |
| 84 | - mr |
| 85 | - ms |
| 86 | - mk |
| 87 | - mg |
| 88 | - mt |
| 89 | - mn |
| 90 | - mi |
| 91 | - my |
| 92 | - zh |
| 93 | - nl |
| 94 | - 'no' |
| 95 | - 'no' |
| 96 | - ne |
| 97 | - ny |
| 98 | - oc |
| 99 | - om |
| 100 | - or |
| 101 | - os |
| 102 | - pa |
| 103 | - pl |
| 104 | - pt |
| 105 | - ms |
| 106 | - ps |
| 107 | - qu |
| 108 | - qu |
| 109 | - qu |
| 110 | - qu |
| 111 | - qu |
| 112 | - qu |
| 113 | - qu |
| 114 | - qu |
| 115 | - qu |
| 116 | - qu |
| 117 | - qu |
| 118 | - qu |
| 119 | - qu |
| 120 | - qu |
| 121 | - qu |
| 122 | - qu |
| 123 | - qu |
| 124 | - qu |
| 125 | - qu |
| 126 | - qu |
| 127 | - qu |
| 128 | - qu |
| 129 | - ro |
| 130 | - rn |
| 131 | - ru |
| 132 | - sg |
| 133 | - sk |
| 134 | - sl |
| 135 | - sm |
| 136 | - sn |
| 137 | - sd |
| 138 | - so |
| 139 | - es |
| 140 | - sq |
| 141 | - su |
| 142 | - sv |
| 143 | - sw |
| 144 | - ta |
| 145 | - tt |
| 146 | - te |
| 147 | - tg |
| 148 | - tl |
| 149 | - th |
| 150 | - ti |
| 151 | - ts |
| 152 | - tr |
| 153 | - uk |
| 154 | - ms |
| 155 | - vi |
| 156 | - wo |
| 157 | - xh |
| 158 | - ms |
| 159 | - yo |
| 160 | - ms |
| 161 | - zu |
| 162 | - za |
| 163 | license: cc-by-nc-4.0 |
| 164 | datasets: |
| 165 | - google/fleurs |
| 166 | metrics: |
| 167 | - acc |
| 168 | --- |
| 169 | |
| 170 | # Massively Multilingual Speech (MMS) - Finetuned LID |
| 171 | |
| 172 | This checkpoint is a model fine-tuned for speech language identification (LID) and part of Facebook's [Massive Multilingual Speech project](https://research.facebook.com/publications/scaling-speech-technology-to-1000-languages/). |
| 173 | This checkpoint is based on the [Wav2Vec2 architecture](https://huggingface.co/docs/transformers/model_doc/wav2vec2) and classifies raw audio input to a probability distribution over 126 output classes (each class representing a language). |
| 174 | The checkpoint consists of **1 billion parameters** and has been fine-tuned from [facebook/mms-1b](https://huggingface.co/facebook/mms-1b) on 126 languages. |
| 175 | |
| 176 | ## Table Of Content |
| 177 | |
| 178 | - [Example](#example) |
| 179 | - [Supported Languages](#supported-languages) |
| 180 | - [Model details](#model-details) |
| 181 | - [Additional links](#additional-links) |
| 182 | |
| 183 | ## Example |
| 184 | |
| 185 | This MMS checkpoint can be used with [Transformers](https://github.com/huggingface/transformers) to identify |
| 186 | the spoken language of an audio. It can recognize the [following 126 languages](#supported-languages). |
| 187 | |
| 188 | Let's look at a simple example. |
| 189 | |
| 190 | First, we install transformers and some other libraries |
| 191 | ``` |
| 192 | pip install torch accelerate torchaudio datasets |
| 193 | pip install --upgrade transformers |
| 194 | ```` |
| 195 | |
| 196 | **Note**: In order to use MMS you need to have at least `transformers >= 4.30` installed. If the `4.30` version |
| 197 | is not yet available [on PyPI](https://pypi.org/project/transformers/) make sure to install `transformers` from |
| 198 | source: |
| 199 | ``` |
| 200 | pip install git+https://github.com/huggingface/transformers.git |
| 201 | ``` |
| 202 | |
| 203 | Next, we load a couple of audio samples via `datasets`. Make sure that the audio data is sampled to 16000 kHz. |
| 204 | |
| 205 | ```py |
| 206 | from datasets import load_dataset, Audio |
| 207 | |
| 208 | # English |
| 209 | stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True) |
| 210 | stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000)) |
| 211 | en_sample = next(iter(stream_data))["audio"]["array"] |
| 212 | |
| 213 | # Arabic |
| 214 | stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "ar", split="test", streaming=True) |
| 215 | stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000)) |
| 216 | ar_sample = next(iter(stream_data))["audio"]["array"] |
| 217 | ``` |
| 218 | |
| 219 | Next, we load the model and processor |
| 220 | |
| 221 | ```py |
| 222 | from transformers import Wav2Vec2ForSequenceClassification, AutoFeatureExtractor |
| 223 | import torch |
| 224 | |
| 225 | model_id = "facebook/mms-lid-126" |
| 226 | |
| 227 | processor = AutoFeatureExtractor.from_pretrained(model_id) |
| 228 | model = Wav2Vec2ForSequenceClassification.from_pretrained(model_id) |
| 229 | ``` |
| 230 | |
| 231 | Now we process the audio data, pass the processed audio data to the model to classify it into a language, just like we usually do for Wav2Vec2 audio classification models such as [ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition](https://huggingface.co/harshit345/xlsr-wav2vec-speech-emotion-recognition) |
| 232 | |
| 233 | ```py |
| 234 | # English |
| 235 | inputs = processor(en_sample, sampling_rate=16_000, return_tensors="pt") |
| 236 | |
| 237 | with torch.no_grad(): |
| 238 | outputs = model(**inputs).logits |
| 239 | |
| 240 | lang_id = torch.argmax(outputs, dim=-1)[0].item() |
| 241 | detected_lang = model.config.id2label[lang_id] |
| 242 | # 'eng' |
| 243 | |
| 244 | # Arabic |
| 245 | inputs = processor(ar_sample, sampling_rate=16_000, return_tensors="pt") |
| 246 | |
| 247 | with torch.no_grad(): |
| 248 | outputs = model(**inputs).logits |
| 249 | |
| 250 | lang_id = torch.argmax(outputs, dim=-1)[0].item() |
| 251 | detected_lang = model.config.id2label[lang_id] |
| 252 | # 'ara' |
| 253 | ``` |
| 254 | |
| 255 | To see all the supported languages of a checkpoint, you can print out the language ids as follows: |
| 256 | ```py |
| 257 | processor.id2label.values() |
| 258 | ``` |
| 259 | |
| 260 | For more details, about the architecture please have a look at [the official docs](https://huggingface.co/docs/transformers/main/en/model_doc/mms). |
| 261 | |
| 262 | ## Supported Languages |
| 263 | |
| 264 | This model supports 126 languages. Unclick the following to toogle all supported languages of this checkpoint in [ISO 639-3 code](https://en.wikipedia.org/wiki/ISO_639-3). |
| 265 | You can find more details about the languages and their ISO 649-3 codes in the [MMS Language Coverage Overview](https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html). |
| 266 | <details> |
| 267 | <summary>Click to toggle</summary> |
| 268 | |
| 269 | - ara |
| 270 | - cmn |
| 271 | - eng |
| 272 | - spa |
| 273 | - fra |
| 274 | - mlg |
| 275 | - swe |
| 276 | - por |
| 277 | - vie |
| 278 | - ful |
| 279 | - sun |
| 280 | - asm |
| 281 | - ben |
| 282 | - zlm |
| 283 | - kor |
| 284 | - ind |
| 285 | - hin |
| 286 | - tuk |
| 287 | - urd |
| 288 | - aze |
| 289 | - slv |
| 290 | - mon |
| 291 | - hau |
| 292 | - tel |
| 293 | - swh |
| 294 | - bod |
| 295 | - rus |
| 296 | - tur |
| 297 | - heb |
| 298 | - mar |
| 299 | - som |
| 300 | - tgl |
| 301 | - tat |
| 302 | - tha |
| 303 | - cat |
| 304 | - ron |
| 305 | - mal |
| 306 | - bel |
| 307 | - pol |
| 308 | - yor |
| 309 | - nld |
| 310 | - bul |
| 311 | - hat |
| 312 | - afr |
| 313 | - isl |
| 314 | - amh |
| 315 | - tam |
| 316 | - hun |
| 317 | - hrv |
| 318 | - lit |
| 319 | - cym |
| 320 | - fas |
| 321 | - mkd |
| 322 | - ell |
| 323 | - bos |
| 324 | - deu |
| 325 | - sqi |
| 326 | - jav |
| 327 | - nob |
| 328 | - uzb |
| 329 | - snd |
| 330 | - lat |
| 331 | - nya |
| 332 | - grn |
| 333 | - mya |
| 334 | - orm |
| 335 | - lin |
| 336 | - hye |
| 337 | - yue |
| 338 | - pan |
| 339 | - jpn |
| 340 | - kaz |
| 341 | - npi |
| 342 | - kat |
| 343 | - guj |
| 344 | - kan |
| 345 | - tgk |
| 346 | - ukr |
| 347 | - ces |
| 348 | - lav |
| 349 | - bak |
| 350 | - khm |
| 351 | - fao |
| 352 | - glg |
| 353 | - ltz |
| 354 | - lao |
| 355 | - mlt |
| 356 | - sin |
| 357 | - sna |
| 358 | - ita |
| 359 | - srp |
| 360 | - mri |
| 361 | - nno |
| 362 | - pus |
| 363 | - eus |
| 364 | - ory |
| 365 | - lug |
| 366 | - bre |
| 367 | - luo |
| 368 | - slk |
| 369 | - fin |
| 370 | - dan |
| 371 | - yid |
| 372 | - est |
| 373 | - ceb |
| 374 | - war |
| 375 | - san |
| 376 | - kir |
| 377 | - oci |
| 378 | - wol |
| 379 | - haw |
| 380 | - kam |
| 381 | - umb |
| 382 | - xho |
| 383 | - epo |
| 384 | - zul |
| 385 | - ibo |
| 386 | - abk |
| 387 | - ckb |
| 388 | - nso |
| 389 | - gle |
| 390 | - kea |
| 391 | - ast |
| 392 | - sco |
| 393 | - glv |
| 394 | - ina |
| 395 | |
| 396 | </details> |
| 397 | |
| 398 | ## Model details |
| 399 | |
| 400 | - **Developed by:** Vineel Pratap et al. |
| 401 | - **Model type:** Multi-Lingual Automatic Speech Recognition model |
| 402 | - **Language(s):** 126 languages, see [supported languages](#supported-languages) |
| 403 | - **License:** CC-BY-NC 4.0 license |
| 404 | - **Num parameters**: 1 billion |
| 405 | - **Audio sampling rate**: 16,000 kHz |
| 406 | - **Cite as:** |
| 407 | |
| 408 | @article{pratap2023mms, |
| 409 | title={Scaling Speech Technology to 1,000+ Languages}, |
| 410 | author={Vineel Pratap and Andros Tjandra and Bowen Shi and Paden Tomasello and Arun Babu and Sayani Kundu and Ali Elkahky and Zhaoheng Ni and Apoorv Vyas and Maryam Fazel-Zarandi and Alexei Baevski and Yossi Adi and Xiaohui Zhang and Wei-Ning Hsu and Alexis Conneau and Michael Auli}, |
| 411 | journal={arXiv}, |
| 412 | year={2023} |
| 413 | } |
| 414 | |
| 415 | ## Additional Links |
| 416 | |
| 417 | - [Blog post](https://ai.facebook.com/blog/multilingual-model-speech-recognition/) |
| 418 | - [Transformers documentation](https://huggingface.co/docs/transformers/main/en/model_doc/mms). |
| 419 | - [Paper](https://arxiv.org/abs/2305.13516) |
| 420 | - [GitHub Repository](https://github.com/facebookresearch/fairseq/tree/main/examples/mms#asr) |
| 421 | - [Other **MMS** checkpoints](https://huggingface.co/models?other=mms) |
| 422 | - MMS base checkpoints: |
| 423 | - [facebook/mms-1b](https://huggingface.co/facebook/mms-1b) |
| 424 | - [facebook/mms-300m](https://huggingface.co/facebook/mms-300m) |
| 425 | - [Official Space](https://huggingface.co/spaces/facebook/MMS) |
| 426 | |