README.md · mDeBERTa-v3-base-xnli-multilingual-nli-2mil7

1

---

2

language:

3

- multilingual

4

- zh

5

- ja

6

- ar

7

- ko

8

- de

9

- fr

10

- es

11

- pt

12

- hi

13

- id

14

- it

15

- tr

16

- ru

17

- bn

18

- ur

19

- mr

20

- ta

21

- vi

22

- fa

23

- pl

24

- uk

25

- nl

26

- sv

27

- he

28

- sw

29

- ps

30

license: mit

31

tags:

32

- zero-shot-classification

33

- text-classification

34

- nli

35

- pytorch

36

datasets:

37

- MoritzLaurer/multilingual-NLI-26lang-2mil7

38

- xnli

39

- multi_nli

40

- facebook/anli

41

- fever

42

- lingnli

43

- alisawuffles/WANLI

44

metrics:

45

- accuracy

46

pipeline_tag: zero-shot-classification

47

widget:

48

- text: Angela Merkel ist eine Politikerin in Deutschland und Vorsitzende der CDU

49

candidate_labels: politics, economy, entertainment, environment

50

model-index:

51

- name: DeBERTa-v3-base-xnli-multilingual-nli-2mil7

52

results:

53

- task:

54

type: text-classification

55

name: Natural Language Inference

56

dataset:

57

name: MultiNLI-matched

58

type: multi_nli

59

split: validation_matched

60

metrics:

61

- type: accuracy

62

value: 0,857

63

verified: false

64

- task:

65

type: text-classification

66

name: Natural Language Inference

67

dataset:

68

name: MultiNLI-mismatched

69

type: multi_nli

70

split: validation_mismatched

71

metrics:

72

- type: accuracy

73

value: 0,856

74

verified: false

75

- task:

76

type: text-classification

77

name: Natural Language Inference

78

dataset:

79

name: ANLI-all

80

type: anli

81

split: test_r1+test_r2+test_r3

82

metrics:

83

- type: accuracy

84

value: 0,537

85

verified: false

86

- task:

87

type: text-classification

88

name: Natural Language Inference

89

dataset:

90

name: ANLI-r3

91

type: anli

92

split: test_r3

93

metrics:

94

- type: accuracy

95

value: 0,497

96

verified: false

97

- task:

98

type: text-classification

99

name: Natural Language Inference

100

dataset:

101

name: WANLI

102

type: alisawuffles/WANLI

103

split: test

104

metrics:

105

- type: accuracy

106

value: 0,732

107

verified: false

108

- task:

109

type: text-classification

110

name: Natural Language Inference

111

dataset:

112

name: LingNLI

113

type: lingnli

114

split: test

115

metrics:

116

- type: accuracy

117

value: 0,788

118

verified: false

119

- task:

120

type: text-classification

121

name: Natural Language Inference

122

dataset:

123

name: fever-nli

124

type: fever-nli

125

split: test

126

metrics:

127

- type: accuracy

128

value: 0,761

129

verified: false

130

---

131

# Model card for mDeBERTa-v3-base-xnli-multilingual-nli-2mil7

132

133

## Model description

134

135

This multilingual model can perform natural language inference (NLI) on 100 languages and is therefore also suitable for multilingual zero-shot classification. The underlying mDeBERTa-v3-base model was pre-trained by Microsoft on the [CC100 multilingual dataset](https://huggingface.co/datasets/cc100) with 100 languages. The model was then fine-tuned on the [XNLI dataset](https://huggingface.co/datasets/xnli) and on the [multilingual-NLI-26lang-2mil7 dataset](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7). Both datasets contain more than 2.7 million hypothesis-premise pairs in 27 languages spoken by more than 4 billion people.

136

137

As of December 2021, mDeBERTa-v3-base is the best performing multilingual base-sized transformer model introduced by Microsoft in [this paper](https://arxiv.org/pdf/2111.09543.pdf).

138

139

140

### How to use the model

141

#### Simple zero-shot classification pipeline

142

```python

143

from transformers import pipeline

144

classifier = pipeline("zero-shot-classification", model="MoritzLaurer/mDeBERTa-v3-base-mnli-xnli")

145

sequence_to_classify = "Angela Merkel ist eine Politikerin in Deutschland und Vorsitzende der CDU"

146

candidate_labels = ["politics", "economy", "entertainment", "environment"]

147

output = classifier(sequence_to_classify, candidate_labels, multi_label=False)

148

print(output)

149

```

150

#### NLI use-case

151

```python

152

from transformers import AutoTokenizer, AutoModelForSequenceClassification

153

import torch

154

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

155

156

model_name = "MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7"

157

tokenizer = AutoTokenizer.from_pretrained(model_name)

158

model = AutoModelForSequenceClassification.from_pretrained(model_name)

159

160

premise = "Angela Merkel ist eine Politikerin in Deutschland und Vorsitzende der CDU"

161

hypothesis = "Emmanuel Macron is the President of France"

162

163

input = tokenizer(premise, hypothesis, truncation=True, return_tensors="pt")

164

output = model(input["input_ids"].to(device)) # device = "cuda:0" or "cpu"

165

prediction = torch.softmax(output["logits"][0], -1).tolist()

166

label_names = ["entailment", "neutral", "contradiction"]

167

prediction = {name: round(float(pred) * 100, 1) for pred, name in zip(prediction, label_names)}

168

print(prediction)

169

```

170

171

### Training data

172

This model was trained on the [multilingual-nli-26lang-2mil7 dataset](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7) and the [XNLI](https://huggingface.co/datasets/xnli) validation dataset.

173

174

The multilingual-nli-26lang-2mil7 dataset contains 2 730 000 NLI hypothesis-premise pairs in 26 languages spoken by more than 4 billion people. The dataset contains 105 000 text pairs per language. It is based on the English datasets [MultiNLI](https://huggingface.co/datasets/multi_nli), [Fever-NLI](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md), [ANLI](https://huggingface.co/datasets/anli), [LingNLI](https://arxiv.org/pdf/2104.07179.pdf) and [WANLI](https://huggingface.co/datasets/alisawuffles/WANLI) and was created using the latest open-source machine translation models. The languages in the dataset are: ['ar', 'bn', 'de', 'es', 'fa', 'fr', 'he', 'hi', 'id', 'it', 'ja', 'ko', 'mr', 'nl', 'pl', 'ps', 'pt', 'ru', 'sv', 'sw', 'ta', 'tr', 'uk', 'ur', 'vi', 'zh'] (see [ISO language codes](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes). For more details, see the [datasheet](XXX). In addition, a sample of 105 000 text pairs was also added for English following the same sampling method as the other languages, leading to 27 languages.

175

176

Moreover, for each language a random set of 10% of the hypothesis-premise pairs was added where an English hypothesis was paired with the premise in the other language (and the same for English premises and other language hypotheses). This mix of languages in the text pairs should enable users to formulate a hypothesis in English for a target text in another language.

177

178

The [XNLI](https://huggingface.co/datasets/xnli) validation set consists of 2490 professionally translated texts from English to 14 other languages (37350 texts in total) (see [this paper](https://arxiv.org/pdf/1809.05053.pdf)). Note that XNLI also contains a training set of 14 machine translated versions of the MultiNLI dataset for 14 languages, but this data was excluded due to quality issues with the machine translations from 2018.

179

180

Note that for evaluation purposes, three languages were excluded from the XNLI training data and only included in the test data: ["bg","el","th"]. This was done in order to test the performance of the model on languages it has not seen during NLI fine-tuning on 27 languages, but only during pre-training on 100 languages - see evaluation metrics below.

181

182

The total training dataset had a size of 3 287 280 hypothesis-premise pairs.

183

184

185

### Training procedure

186

mDeBERTa-v3-base-mnli-xnli was trained using the Hugging Face trainer with the following hyperparameters.

187

188

```

189

training_args = TrainingArguments(

190

num_train_epochs=3, # total number of training epochs

191

learning_rate=2e-05,

192

per_device_train_batch_size=32, # batch size per device during training

193

gradient_accumulation_steps=2, # to double the effective batch size for

194

warmup_ratio=0.06, # number of warmup steps for learning rate scheduler

195

weight_decay=0.01, # strength of weight decay

196

fp16=False

197

)

198

```

199

200

### Eval results

201

The model was evaluated on the XNLI test set in 15 languages (5010 texts per language, 75150 in total) and the English test sets of [MultiNLI](https://huggingface.co/datasets/multi_nli), [Fever-NLI](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md), [ANLI](https://huggingface.co/datasets/anli), [LingNLI](https://arxiv.org/pdf/2104.07179.pdf) and [WANLI](https://huggingface.co/datasets/alisawuffles/WANLI) . Note that multilingual NLI models are capable of classifying NLI texts without receiving NLI training data in the specific language (cross-lingual transfer). This means that the model is also able to do NLI on the other 73 languages mDeBERTa was pre-trained on, but performance is most likely lower than for those languages seen during NLI fine-tuning. The performance on the languages ["bg","el","th"] in the table below is a good indicated of this cross-lingual transfer, as these languages were not included in the training data.

202

203

|XNLI subsets|ar|bg|de|el|en|es|fr|hi|ru|sw|th|tr|ur|vi|zh|

204

| :---: |:---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |

205

|Accuracy|0.794|0.822|0.824|0.809|0.871|0.832|0.823|0.769|0.803|0.746|0.786|0.792|0.744|0.793|0.803|

206

|Speed (text/sec, A100-GPU)|1344.0|1355.0|1472.0|1149.0|1697.0|1446.0|1278.0|1115.0|1380.0|1463.0|1713.0|1594.0|1189.0|877.0|1887.0|

207

208

209

| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |

210

|Accuracy|0.857|0.856|0.537|0.497|0.761|0.788|0.732|0.794|

211

|Speed (text/sec, A100-GPU)|1000.0|1009.0|794.0|672.0|374.0|1177.0|1468.0|

212

213

214

Also note that if other multilingual models on the model hub claim performance of around 90% on languages other than English, the authors have most likely made a mistake during testing since non of the latest papers shows a multilingual average performance of more than a few points above 80% on XNLI (see [here](https://arxiv.org/pdf/2111.09543.pdf) or [here](https://arxiv.org/pdf/1911.02116.pdf)).

215

216

217

## Limitations and bias

218

Please consult the original DeBERTa-V3 paper and literature on different NLI datasets for potential biases. Moreover, note that the multilingual-nli-26lang-2mil7 dataset was created using machine translation, which reduces the quality of the data for a complex task like NLI. You can inspect the data via the Hugging Face [dataset viewer](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7) for languages you are interested in. Note that grammatical errors introduced by machine translation are less of an issue for zero-shot classification, for which grammar is less important.

219

220

221

## Citation

222

223

If the dataset is useful for you, please cite the following article:

224

```

225

@article{laurer_less_2022,

226

	title = {Less {Annotating}, {More} {Classifying} – {Addressing} the {Data} {Scarcity} {Issue} of {Supervised} {Machine} {Learning} with {Deep} {Transfer} {Learning} and {BERT} - {NLI}},

227

url = {https://osf.io/74b8k},

228

language = {en-us},

229

urldate = {2022-07-28},

230

journal = {Preprint},

231

author = {Laurer, Moritz and Atteveldt, Wouter van and Casas, Andreu Salleras and Welbers, Kasper},

232

month = jun,

233

year = {2022},

234

note = {Publisher: Open Science Framework},

235

}

236

```

237

238

239

## Ideas for cooperation or questions?

240

For updates on new models and datasets, follow me on [Twitter](https://twitter.com/MoritzLaurer).

241

If you have questions or ideas for cooperation, contact me at m{dot}laurer{at}vu{dot}nl or on [LinkedIn](https://www.linkedin.com/in/moritz-laurer/)

242

243

244

## Debugging and issues

245

Note that DeBERTa-v3 was released in late 2021 and older versions of HF Transformers seem to have issues running the model (e.g. resulting in an issue with the tokenizer). Using Transformers==4.13 or higher might solve some issues. Note that mDeBERTa currently does not support FP16, see here: https://github.com/microsoft/DeBERTa/issues/77

246

247

248

249