README.md
9.3 KB · 152 lines · markdown Raw
1 ---
2 language: en
3 tags:
4 - text-classification
5 - pytorch
6 - roberta
7 - emotions
8 - multi-class-classification
9 - multi-label-classification
10 datasets:
11 - go_emotions
12 license: mit
13 widget:
14 - text: I am not having a great day.
15 ---
16
17 #### Overview
18
19 Model trained from [roberta-base](https://huggingface.co/roberta-base) on the [go_emotions](https://huggingface.co/datasets/go_emotions) dataset for multi-label classification.
20
21 ##### ONNX version also available
22
23 A version of this model in ONNX format (including an INT8 quantized ONNX version) is now available at [https://huggingface.co/SamLowe/roberta-base-go_emotions-onnx](https://huggingface.co/SamLowe/roberta-base-go_emotions-onnx). These are faster for inference, esp for smaller batch sizes, massively reduce the size of the dependencies required for inference, make inference of the model more multi-platform, and in the case of the quantized version reduce the model file/download size by 75% whilst retaining almost all the accuracy if you only need inference.
24
25 #### Dataset used for the model
26
27 [go_emotions](https://huggingface.co/datasets/go_emotions) is based on Reddit data and has 28 labels. It is a multi-label dataset where one or multiple labels may apply for any given input text, hence this model is a multi-label classification model with 28 'probability' float outputs for any given input text. Typically a threshold of 0.5 is applied to the probabilities for the prediction for each label.
28
29 #### How the model was created
30
31 The model was trained using `AutoModelForSequenceClassification.from_pretrained` with `problem_type="multi_label_classification"` for 3 epochs with a learning rate of 2e-5 and weight decay of 0.01.
32
33 #### Inference
34
35 There are multiple ways to use this model in Huggingface Transformers. Possibly the simplest is using a pipeline:
36
37 ```python
38 from transformers import pipeline
39
40 classifier = pipeline(task="text-classification", model="SamLowe/roberta-base-go_emotions", top_k=None)
41
42 sentences = ["I am not having a great day"]
43
44 model_outputs = classifier(sentences)
45 print(model_outputs[0])
46 # produces a list of dicts for each of the labels
47 ```
48
49 #### Evaluation / metrics
50
51 Evaluation of the model is available at
52
53 - https://github.com/samlowe/go_emotions-dataset/blob/main/eval-roberta-base-go_emotions.ipynb
54
55 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/samlowe/go_emotions-dataset/blob/main/eval-roberta-base-go_emotions.ipynb)
56
57 ##### Summary
58
59 As provided in the above notebook, evaluation of the multi-label output (of the 28 dim output via a threshold of 0.5 to binarize each) using the dataset test split gives:
60
61 - Accuracy: 0.474
62 - Precision: 0.575
63 - Recall: 0.396
64 - F1: 0.450
65
66 But the metrics are more meaningful when measured per label given the multi-label nature (each label is effectively an independent binary classification) and the fact that there is drastically different representations of the labels in the dataset.
67
68 With a threshold of 0.5 applied to binarize the model outputs, as per the above notebook, the metrics per label are:
69
70 | | accuracy | precision | recall | f1 | mcc | support | threshold |
71 | -------------- | -------- | --------- | ------ | ----- | ----- | ------- | --------- |
72 | admiration | 0.946 | 0.725 | 0.675 | 0.699 | 0.670 | 504 | 0.5 |
73 | amusement | 0.982 | 0.790 | 0.871 | 0.829 | 0.821 | 264 | 0.5 |
74 | anger | 0.970 | 0.652 | 0.379 | 0.479 | 0.483 | 198 | 0.5 |
75 | annoyance | 0.940 | 0.472 | 0.159 | 0.238 | 0.250 | 320 | 0.5 |
76 | approval | 0.942 | 0.609 | 0.302 | 0.404 | 0.403 | 351 | 0.5 |
77 | caring | 0.973 | 0.448 | 0.319 | 0.372 | 0.364 | 135 | 0.5 |
78 | confusion | 0.972 | 0.500 | 0.431 | 0.463 | 0.450 | 153 | 0.5 |
79 | curiosity | 0.950 | 0.537 | 0.356 | 0.428 | 0.412 | 284 | 0.5 |
80 | desire | 0.987 | 0.630 | 0.410 | 0.496 | 0.502 | 83 | 0.5 |
81 | disappointment | 0.974 | 0.625 | 0.199 | 0.302 | 0.343 | 151 | 0.5 |
82 | disapproval | 0.950 | 0.494 | 0.307 | 0.379 | 0.365 | 267 | 0.5 |
83 | disgust | 0.982 | 0.707 | 0.333 | 0.453 | 0.478 | 123 | 0.5 |
84 | embarrassment | 0.994 | 0.750 | 0.243 | 0.367 | 0.425 | 37 | 0.5 |
85 | excitement | 0.983 | 0.603 | 0.340 | 0.435 | 0.445 | 103 | 0.5 |
86 | fear | 0.992 | 0.758 | 0.603 | 0.671 | 0.672 | 78 | 0.5 |
87 | gratitude | 0.990 | 0.960 | 0.881 | 0.919 | 0.914 | 352 | 0.5 |
88 | grief | 0.999 | 0.000 | 0.000 | 0.000 | 0.000 | 6 | 0.5 |
89 | joy | 0.978 | 0.647 | 0.559 | 0.600 | 0.590 | 161 | 0.5 |
90 | love | 0.982 | 0.773 | 0.832 | 0.802 | 0.793 | 238 | 0.5 |
91 | nervousness | 0.996 | 0.600 | 0.130 | 0.214 | 0.278 | 23 | 0.5 |
92 | optimism | 0.972 | 0.667 | 0.376 | 0.481 | 0.488 | 186 | 0.5 |
93 | pride | 0.997 | 0.000 | 0.000 | 0.000 | 0.000 | 16 | 0.5 |
94 | realization | 0.974 | 0.541 | 0.138 | 0.220 | 0.264 | 145 | 0.5 |
95 | relief | 0.998 | 0.000 | 0.000 | 0.000 | 0.000 | 11 | 0.5 |
96 | remorse | 0.991 | 0.553 | 0.750 | 0.636 | 0.640 | 56 | 0.5 |
97 | sadness | 0.977 | 0.621 | 0.494 | 0.550 | 0.542 | 156 | 0.5 |
98 | surprise | 0.981 | 0.750 | 0.404 | 0.525 | 0.542 | 141 | 0.5 |
99 | neutral | 0.782 | 0.694 | 0.604 | 0.646 | 0.492 | 1787 | 0.5 |
100
101 Optimizing the threshold per label for the one that gives the optimum F1 metrics gives slightly better metrics - sacrificing some precision for a greater gain in recall, hence to the benefit of F1 (how this was done is shown in the above notebook):
102
103 | | accuracy | precision | recall | f1 | mcc | support | threshold |
104 | -------------- | -------- | --------- | ------ | ----- | ----- | ------- | --------- |
105 | admiration | 0.940 | 0.651 | 0.776 | 0.708 | 0.678 | 504 | 0.25 |
106 | amusement | 0.982 | 0.781 | 0.890 | 0.832 | 0.825 | 264 | 0.45 |
107 | anger | 0.959 | 0.454 | 0.601 | 0.517 | 0.502 | 198 | 0.15 |
108 | annoyance | 0.864 | 0.243 | 0.619 | 0.349 | 0.328 | 320 | 0.10 |
109 | approval | 0.926 | 0.432 | 0.442 | 0.437 | 0.397 | 351 | 0.30 |
110 | caring | 0.972 | 0.426 | 0.385 | 0.405 | 0.391 | 135 | 0.40 |
111 | confusion | 0.974 | 0.548 | 0.412 | 0.470 | 0.462 | 153 | 0.55 |
112 | curiosity | 0.943 | 0.473 | 0.711 | 0.568 | 0.552 | 284 | 0.25 |
113 | desire | 0.985 | 0.518 | 0.530 | 0.524 | 0.516 | 83 | 0.25 |
114 | disappointment | 0.974 | 0.562 | 0.298 | 0.390 | 0.398 | 151 | 0.40 |
115 | disapproval | 0.941 | 0.414 | 0.468 | 0.439 | 0.409 | 267 | 0.30 |
116 | disgust | 0.978 | 0.523 | 0.463 | 0.491 | 0.481 | 123 | 0.20 |
117 | embarrassment | 0.994 | 0.567 | 0.459 | 0.507 | 0.507 | 37 | 0.10 |
118 | excitement | 0.981 | 0.500 | 0.417 | 0.455 | 0.447 | 103 | 0.35 |
119 | fear | 0.991 | 0.712 | 0.667 | 0.689 | 0.685 | 78 | 0.40 |
120 | gratitude | 0.990 | 0.957 | 0.889 | 0.922 | 0.917 | 352 | 0.45 |
121 | grief | 0.999 | 0.333 | 0.333 | 0.333 | 0.333 | 6 | 0.05 |
122 | joy | 0.978 | 0.623 | 0.646 | 0.634 | 0.623 | 161 | 0.40 |
123 | love | 0.982 | 0.740 | 0.899 | 0.812 | 0.807 | 238 | 0.25 |
124 | nervousness | 0.996 | 0.571 | 0.348 | 0.432 | 0.444 | 23 | 0.25 |
125 | optimism | 0.971 | 0.580 | 0.565 | 0.572 | 0.557 | 186 | 0.20 |
126 | pride | 0.998 | 0.875 | 0.438 | 0.583 | 0.618 | 16 | 0.10 |
127 | realization | 0.961 | 0.270 | 0.262 | 0.266 | 0.246 | 145 | 0.15 |
128 | relief | 0.992 | 0.152 | 0.636 | 0.246 | 0.309 | 11 | 0.05 |
129 | remorse | 0.991 | 0.541 | 0.946 | 0.688 | 0.712 | 56 | 0.10 |
130 | sadness | 0.977 | 0.599 | 0.583 | 0.591 | 0.579 | 156 | 0.40 |
131 | surprise | 0.977 | 0.543 | 0.674 | 0.601 | 0.593 | 141 | 0.15 |
132 | neutral | 0.758 | 0.598 | 0.810 | 0.688 | 0.513 | 1787 | 0.25 |
133
134 This improves the overall metrics:
135
136 - Precision: 0.542
137 - Recall: 0.577
138 - F1: 0.541
139
140 Or if calculated weighted by the relative size of the support of each label:
141
142 - Precision: 0.572
143 - Recall: 0.677
144 - F1: 0.611
145
146 #### Commentary on the dataset
147
148 Some labels (E.g. gratitude) when considered independently perform very strongly with F1 exceeding 0.9, whilst others (E.g. relief) perform very poorly.
149
150 This is a challenging dataset. Labels such as relief do have much fewer examples in the training data (less than 100 out of the 40k+, and only 11 in the test split).
151
152 But there is also some ambiguity and/or labelling errors visible in the training data of go_emotions that is suspected to constrain the performance. Data cleaning on the dataset to reduce some of the mistakes, ambiguity, conflicts and duplication in the labelling would produce a higher performing model.