README.md
6.7 KB · 161 lines · markdown Raw
1
2 ---
3 language:
4 - multilingual
5 - en
6 - ar
7 - bg
8 - de
9 - el
10 - es
11 - fr
12 - hi
13 - ru
14 - sw
15 - th
16 - tr
17 - ur
18 - vi
19 - zh
20 license: mit
21 tags:
22 - zero-shot-classification
23 - text-classification
24 - nli
25 - pytorch
26 metrics:
27 - accuracy
28 datasets:
29 - multi_nli
30 - xnli
31 pipeline_tag: zero-shot-classification
32 widget:
33 - text: "Angela Merkel ist eine Politikerin in Deutschland und Vorsitzende der CDU"
34 candidate_labels: "politics, economy, entertainment, environment"
35 ---
36
37
38 ---
39 # Multilingual MiniLMv2-L12-mnli-xnli
40 ## Model description
41 This multilingual model can perform natural language inference (NLI) on 100+ languages and is therefore also
42 suitable for multilingual zero-shot classification. The underlying multilingual-MiniLM-L12 model was created
43 by Microsoft and was distilled from XLM-RoBERTa-large (see details [in the original paper](https://arxiv.org/pdf/2002.10957.pdf)
44 and newer information in [this repo](https://github.com/microsoft/unilm/tree/master/minilm)).
45 The model was then fine-tuned on the [XNLI dataset](https://huggingface.co/datasets/xnli), which contains hypothesis-premise pairs from 15 languages,
46 as well as the English [MNLI dataset](https://huggingface.co/datasets/multi_nli).
47
48 The main advantage of distilled models is that they are smaller (faster inference, lower memory requirements) than their teachers (XLM-RoBERTa-large).
49 The disadvantage is that they lose some of the performance of their larger teachers.
50
51 For highest inference speed, I recommend using the [6-layer model](https://huggingface.co/MoritzLaurer/multilingual-MiniLMv2-L6-mnli-xnli)
52 (the model on this page has 12 layers and is slower). For higher performance I recommend
53 [mDeBERTa-v3-base-mnli-xnli](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) (as of 14.02.2023).
54
55 ### How to use the model
56 #### Simple zero-shot classification pipeline
57 ```python
58 from transformers import pipeline
59 classifier = pipeline("zero-shot-classification", model="MoritzLaurer/multilingual-MiniLMv2-L12-mnli-xnli")
60
61 sequence_to_classify = "Angela Merkel ist eine Politikerin in Deutschland und Vorsitzende der CDU"
62 candidate_labels = ["politics", "economy", "entertainment", "environment"]
63 output = classifier(sequence_to_classify, candidate_labels, multi_label=False)
64 print(output)
65 ```
66 #### NLI use-case
67 ```python
68 from transformers import AutoTokenizer, AutoModelForSequenceClassification
69 import torch
70 device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
71
72 model_name = "MoritzLaurer/multilingual-MiniLMv2-L12-mnli-xnli"
73 tokenizer = AutoTokenizer.from_pretrained(model_name)
74 model = AutoModelForSequenceClassification.from_pretrained(model_name)
75
76 premise = "Angela Merkel ist eine Politikerin in Deutschland und Vorsitzende der CDU"
77 hypothesis = "Emmanuel Macron is the President of France"
78
79 input = tokenizer(premise, hypothesis, truncation=True, return_tensors="pt")
80 output = model(input["input_ids"].to(device)) # device = "cuda:0" or "cpu"
81 prediction = torch.softmax(output["logits"][0], -1).tolist()
82 label_names = ["entailment", "neutral", "contradiction"]
83 prediction = {name: round(float(pred) * 100, 1) for pred, name in zip(prediction, label_names)}
84 print(prediction)
85 ```
86
87 ### Training data
88 This model was trained on the XNLI development dataset and the MNLI train dataset.
89 The XNLI development set consists of 2490 professionally translated texts from English
90 to 14 other languages (37350 texts in total) (see [this paper](https://arxiv.org/pdf/1809.05053.pdf)).
91 Note that the XNLI contains a training set of 15 machine translated versions of the MNLI dataset for 15 languages,
92 but due to quality issues with these machine translations, this model was only trained on the professional translations
93 from the XNLI development set and the original English MNLI training set (392 702 texts).
94 Not using machine translated texts can avoid overfitting the model to the 15 languages;
95 avoids catastrophic forgetting of the other languages it was pre-trained on;
96 and significantly reduces training costs.
97
98 ### Training procedure
99 The model was trained using the Hugging Face trainer with the following hyperparameters.
100 The exact underlying model is [mMiniLMv2-L12-H384-distilled-from-XLMR-Large](https://huggingface.co/nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large).
101 ```
102 training_args = TrainingArguments(
103 num_train_epochs=3, # total number of training epochs
104 learning_rate=4e-05,
105 per_device_train_batch_size=64, # batch size per device during training
106 per_device_eval_batch_size=120, # batch size for evaluation
107 warmup_ratio=0.06, # number of warmup steps for learning rate scheduler
108 weight_decay=0.01, # strength of weight decay
109 )
110 ```
111
112 ### Eval results
113 The model was evaluated on the XNLI test set on 15 languages (5010 texts per language, 75150 in total).
114 Note that multilingual NLI models are capable of classifying NLI texts without receiving NLI training data
115 in the specific language (cross-lingual transfer). This means that the model is also able of doing NLI on
116 the other languages it was training on, but performance is most likely lower than for those languages available in XNLI.
117
118 The average XNLI performance of multilingual-MiniLM-L12 reported in the paper is 0.711 ([see table 11](https://arxiv.org/pdf/2002.10957.pdf)).
119 This reimplementation has an average performance of 0.75.
120 This increase in performance is probably thanks to the addition of MNLI in the training data and this model was distilled from
121 XLM-RoBERTa-large instead of -base (multilingual-MiniLM-L12-v2).
122
123
124
125 |Datasets|avg_xnli|ar|bg|de|el|en|es|fr|hi|ru|sw|th|tr|ur|vi|zh|
126 | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
127 |Accuracy|0.75|0.73|0.78|0.762|0.754|0.821|0.779|0.775|0.724|0.76|0.689|0.738|0.732|0.7|0.762|0.751|
128 |Speed text/sec (A100 GPU, eval_batch=120)|4535.0|4629.0|4417.0|4500.0|3938.0|4959.0|4634.0|4152.0|4190.0|4368.0|4630.0|4698.0|4929.0|4291.0|4420.0|5275.0|
129
130 |Datasets|mnli_m|mnli_mm|
131 | :---: | :---: | :---: |
132 |Accuracy|0.818|0.831|
133 |Speed text/sec (A100 GPU, eval_batch=120)|2912.0|2902.0|
134
135
136
137 ## Limitations and bias
138 Please consult the original paper and literature on different NLI datasets for potential biases.
139
140 ## Citation
141 If you use this model, please cite: Laurer, Moritz, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022.
142 ‘Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI’.
143 Preprint, June. Open Science Framework. https://osf.io/74b8k.
144
145 ## Ideas for cooperation or questions?
146 If you have questions or ideas for cooperation, contact me at m{dot}laurer{at}vu{dot}nl or [LinkedIn](https://www.linkedin.com/in/moritz-laurer/)
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161