README.md
| 1 | --- |
| 2 | tags: |
| 3 | - mteb |
| 4 | - sentence-similarity |
| 5 | - sentence-transformers |
| 6 | - Sentence Transformers |
| 7 | model-index: |
| 8 | - name: gte-small |
| 9 | results: |
| 10 | - task: |
| 11 | type: Classification |
| 12 | dataset: |
| 13 | type: mteb/amazon_counterfactual |
| 14 | name: MTEB AmazonCounterfactualClassification (en) |
| 15 | config: en |
| 16 | split: test |
| 17 | revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
| 18 | metrics: |
| 19 | - type: accuracy |
| 20 | value: 73.22388059701493 |
| 21 | - type: ap |
| 22 | value: 36.09895941426988 |
| 23 | - type: f1 |
| 24 | value: 67.3205651539195 |
| 25 | - task: |
| 26 | type: Classification |
| 27 | dataset: |
| 28 | type: mteb/amazon_polarity |
| 29 | name: MTEB AmazonPolarityClassification |
| 30 | config: default |
| 31 | split: test |
| 32 | revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
| 33 | metrics: |
| 34 | - type: accuracy |
| 35 | value: 91.81894999999999 |
| 36 | - type: ap |
| 37 | value: 88.5240138417305 |
| 38 | - type: f1 |
| 39 | value: 91.80367382706962 |
| 40 | - task: |
| 41 | type: Classification |
| 42 | dataset: |
| 43 | type: mteb/amazon_reviews_multi |
| 44 | name: MTEB AmazonReviewsClassification (en) |
| 45 | config: en |
| 46 | split: test |
| 47 | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| 48 | metrics: |
| 49 | - type: accuracy |
| 50 | value: 48.032 |
| 51 | - type: f1 |
| 52 | value: 47.4490665674719 |
| 53 | - task: |
| 54 | type: Retrieval |
| 55 | dataset: |
| 56 | type: arguana |
| 57 | name: MTEB ArguAna |
| 58 | config: default |
| 59 | split: test |
| 60 | revision: None |
| 61 | metrics: |
| 62 | - type: map_at_1 |
| 63 | value: 30.725 |
| 64 | - type: map_at_10 |
| 65 | value: 46.604 |
| 66 | - type: map_at_100 |
| 67 | value: 47.535 |
| 68 | - type: map_at_1000 |
| 69 | value: 47.538000000000004 |
| 70 | - type: map_at_3 |
| 71 | value: 41.833 |
| 72 | - type: map_at_5 |
| 73 | value: 44.61 |
| 74 | - type: mrr_at_1 |
| 75 | value: 31.223 |
| 76 | - type: mrr_at_10 |
| 77 | value: 46.794000000000004 |
| 78 | - type: mrr_at_100 |
| 79 | value: 47.725 |
| 80 | - type: mrr_at_1000 |
| 81 | value: 47.727000000000004 |
| 82 | - type: mrr_at_3 |
| 83 | value: 42.07 |
| 84 | - type: mrr_at_5 |
| 85 | value: 44.812000000000005 |
| 86 | - type: ndcg_at_1 |
| 87 | value: 30.725 |
| 88 | - type: ndcg_at_10 |
| 89 | value: 55.440999999999995 |
| 90 | - type: ndcg_at_100 |
| 91 | value: 59.134 |
| 92 | - type: ndcg_at_1000 |
| 93 | value: 59.199 |
| 94 | - type: ndcg_at_3 |
| 95 | value: 45.599000000000004 |
| 96 | - type: ndcg_at_5 |
| 97 | value: 50.637 |
| 98 | - type: precision_at_1 |
| 99 | value: 30.725 |
| 100 | - type: precision_at_10 |
| 101 | value: 8.364 |
| 102 | - type: precision_at_100 |
| 103 | value: 0.991 |
| 104 | - type: precision_at_1000 |
| 105 | value: 0.1 |
| 106 | - type: precision_at_3 |
| 107 | value: 18.848000000000003 |
| 108 | - type: precision_at_5 |
| 109 | value: 13.77 |
| 110 | - type: recall_at_1 |
| 111 | value: 30.725 |
| 112 | - type: recall_at_10 |
| 113 | value: 83.64200000000001 |
| 114 | - type: recall_at_100 |
| 115 | value: 99.14699999999999 |
| 116 | - type: recall_at_1000 |
| 117 | value: 99.644 |
| 118 | - type: recall_at_3 |
| 119 | value: 56.543 |
| 120 | - type: recall_at_5 |
| 121 | value: 68.848 |
| 122 | - task: |
| 123 | type: Clustering |
| 124 | dataset: |
| 125 | type: mteb/arxiv-clustering-p2p |
| 126 | name: MTEB ArxivClusteringP2P |
| 127 | config: default |
| 128 | split: test |
| 129 | revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
| 130 | metrics: |
| 131 | - type: v_measure |
| 132 | value: 47.90178078197678 |
| 133 | - task: |
| 134 | type: Clustering |
| 135 | dataset: |
| 136 | type: mteb/arxiv-clustering-s2s |
| 137 | name: MTEB ArxivClusteringS2S |
| 138 | config: default |
| 139 | split: test |
| 140 | revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
| 141 | metrics: |
| 142 | - type: v_measure |
| 143 | value: 40.25728393431922 |
| 144 | - task: |
| 145 | type: Reranking |
| 146 | dataset: |
| 147 | type: mteb/askubuntudupquestions-reranking |
| 148 | name: MTEB AskUbuntuDupQuestions |
| 149 | config: default |
| 150 | split: test |
| 151 | revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
| 152 | metrics: |
| 153 | - type: map |
| 154 | value: 61.720297062897764 |
| 155 | - type: mrr |
| 156 | value: 75.24139295607439 |
| 157 | - task: |
| 158 | type: STS |
| 159 | dataset: |
| 160 | type: mteb/biosses-sts |
| 161 | name: MTEB BIOSSES |
| 162 | config: default |
| 163 | split: test |
| 164 | revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
| 165 | metrics: |
| 166 | - type: cos_sim_pearson |
| 167 | value: 89.43527309184616 |
| 168 | - type: cos_sim_spearman |
| 169 | value: 88.17128615100206 |
| 170 | - type: euclidean_pearson |
| 171 | value: 87.89922623089282 |
| 172 | - type: euclidean_spearman |
| 173 | value: 87.96104039655451 |
| 174 | - type: manhattan_pearson |
| 175 | value: 87.9818290932077 |
| 176 | - type: manhattan_spearman |
| 177 | value: 88.00923426576885 |
| 178 | - task: |
| 179 | type: Classification |
| 180 | dataset: |
| 181 | type: mteb/banking77 |
| 182 | name: MTEB Banking77Classification |
| 183 | config: default |
| 184 | split: test |
| 185 | revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
| 186 | metrics: |
| 187 | - type: accuracy |
| 188 | value: 84.0844155844156 |
| 189 | - type: f1 |
| 190 | value: 84.01485017302213 |
| 191 | - task: |
| 192 | type: Clustering |
| 193 | dataset: |
| 194 | type: mteb/biorxiv-clustering-p2p |
| 195 | name: MTEB BiorxivClusteringP2P |
| 196 | config: default |
| 197 | split: test |
| 198 | revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
| 199 | metrics: |
| 200 | - type: v_measure |
| 201 | value: 38.36574769259432 |
| 202 | - task: |
| 203 | type: Clustering |
| 204 | dataset: |
| 205 | type: mteb/biorxiv-clustering-s2s |
| 206 | name: MTEB BiorxivClusteringS2S |
| 207 | config: default |
| 208 | split: test |
| 209 | revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
| 210 | metrics: |
| 211 | - type: v_measure |
| 212 | value: 35.4857033165287 |
| 213 | - task: |
| 214 | type: Retrieval |
| 215 | dataset: |
| 216 | type: BeIR/cqadupstack |
| 217 | name: MTEB CQADupstackAndroidRetrieval |
| 218 | config: default |
| 219 | split: test |
| 220 | revision: None |
| 221 | metrics: |
| 222 | - type: map_at_1 |
| 223 | value: 30.261 |
| 224 | - type: map_at_10 |
| 225 | value: 42.419000000000004 |
| 226 | - type: map_at_100 |
| 227 | value: 43.927 |
| 228 | - type: map_at_1000 |
| 229 | value: 44.055 |
| 230 | - type: map_at_3 |
| 231 | value: 38.597 |
| 232 | - type: map_at_5 |
| 233 | value: 40.701 |
| 234 | - type: mrr_at_1 |
| 235 | value: 36.91 |
| 236 | - type: mrr_at_10 |
| 237 | value: 48.02 |
| 238 | - type: mrr_at_100 |
| 239 | value: 48.658 |
| 240 | - type: mrr_at_1000 |
| 241 | value: 48.708 |
| 242 | - type: mrr_at_3 |
| 243 | value: 44.945 |
| 244 | - type: mrr_at_5 |
| 245 | value: 46.705000000000005 |
| 246 | - type: ndcg_at_1 |
| 247 | value: 36.91 |
| 248 | - type: ndcg_at_10 |
| 249 | value: 49.353 |
| 250 | - type: ndcg_at_100 |
| 251 | value: 54.456 |
| 252 | - type: ndcg_at_1000 |
| 253 | value: 56.363 |
| 254 | - type: ndcg_at_3 |
| 255 | value: 43.483 |
| 256 | - type: ndcg_at_5 |
| 257 | value: 46.150999999999996 |
| 258 | - type: precision_at_1 |
| 259 | value: 36.91 |
| 260 | - type: precision_at_10 |
| 261 | value: 9.700000000000001 |
| 262 | - type: precision_at_100 |
| 263 | value: 1.557 |
| 264 | - type: precision_at_1000 |
| 265 | value: 0.202 |
| 266 | - type: precision_at_3 |
| 267 | value: 21.078 |
| 268 | - type: precision_at_5 |
| 269 | value: 15.421999999999999 |
| 270 | - type: recall_at_1 |
| 271 | value: 30.261 |
| 272 | - type: recall_at_10 |
| 273 | value: 63.242 |
| 274 | - type: recall_at_100 |
| 275 | value: 84.09100000000001 |
| 276 | - type: recall_at_1000 |
| 277 | value: 96.143 |
| 278 | - type: recall_at_3 |
| 279 | value: 46.478 |
| 280 | - type: recall_at_5 |
| 281 | value: 53.708 |
| 282 | - task: |
| 283 | type: Retrieval |
| 284 | dataset: |
| 285 | type: BeIR/cqadupstack |
| 286 | name: MTEB CQADupstackEnglishRetrieval |
| 287 | config: default |
| 288 | split: test |
| 289 | revision: None |
| 290 | metrics: |
| 291 | - type: map_at_1 |
| 292 | value: 31.145 |
| 293 | - type: map_at_10 |
| 294 | value: 40.996 |
| 295 | - type: map_at_100 |
| 296 | value: 42.266999999999996 |
| 297 | - type: map_at_1000 |
| 298 | value: 42.397 |
| 299 | - type: map_at_3 |
| 300 | value: 38.005 |
| 301 | - type: map_at_5 |
| 302 | value: 39.628 |
| 303 | - type: mrr_at_1 |
| 304 | value: 38.344 |
| 305 | - type: mrr_at_10 |
| 306 | value: 46.827000000000005 |
| 307 | - type: mrr_at_100 |
| 308 | value: 47.446 |
| 309 | - type: mrr_at_1000 |
| 310 | value: 47.489 |
| 311 | - type: mrr_at_3 |
| 312 | value: 44.448 |
| 313 | - type: mrr_at_5 |
| 314 | value: 45.747 |
| 315 | - type: ndcg_at_1 |
| 316 | value: 38.344 |
| 317 | - type: ndcg_at_10 |
| 318 | value: 46.733000000000004 |
| 319 | - type: ndcg_at_100 |
| 320 | value: 51.103 |
| 321 | - type: ndcg_at_1000 |
| 322 | value: 53.075 |
| 323 | - type: ndcg_at_3 |
| 324 | value: 42.366 |
| 325 | - type: ndcg_at_5 |
| 326 | value: 44.242 |
| 327 | - type: precision_at_1 |
| 328 | value: 38.344 |
| 329 | - type: precision_at_10 |
| 330 | value: 8.822000000000001 |
| 331 | - type: precision_at_100 |
| 332 | value: 1.417 |
| 333 | - type: precision_at_1000 |
| 334 | value: 0.187 |
| 335 | - type: precision_at_3 |
| 336 | value: 20.403 |
| 337 | - type: precision_at_5 |
| 338 | value: 14.306 |
| 339 | - type: recall_at_1 |
| 340 | value: 31.145 |
| 341 | - type: recall_at_10 |
| 342 | value: 56.909 |
| 343 | - type: recall_at_100 |
| 344 | value: 75.274 |
| 345 | - type: recall_at_1000 |
| 346 | value: 87.629 |
| 347 | - type: recall_at_3 |
| 348 | value: 43.784 |
| 349 | - type: recall_at_5 |
| 350 | value: 49.338 |
| 351 | - task: |
| 352 | type: Retrieval |
| 353 | dataset: |
| 354 | type: BeIR/cqadupstack |
| 355 | name: MTEB CQADupstackGamingRetrieval |
| 356 | config: default |
| 357 | split: test |
| 358 | revision: None |
| 359 | metrics: |
| 360 | - type: map_at_1 |
| 361 | value: 38.83 |
| 362 | - type: map_at_10 |
| 363 | value: 51.553000000000004 |
| 364 | - type: map_at_100 |
| 365 | value: 52.581 |
| 366 | - type: map_at_1000 |
| 367 | value: 52.638 |
| 368 | - type: map_at_3 |
| 369 | value: 48.112 |
| 370 | - type: map_at_5 |
| 371 | value: 50.095 |
| 372 | - type: mrr_at_1 |
| 373 | value: 44.513999999999996 |
| 374 | - type: mrr_at_10 |
| 375 | value: 54.998000000000005 |
| 376 | - type: mrr_at_100 |
| 377 | value: 55.650999999999996 |
| 378 | - type: mrr_at_1000 |
| 379 | value: 55.679 |
| 380 | - type: mrr_at_3 |
| 381 | value: 52.602000000000004 |
| 382 | - type: mrr_at_5 |
| 383 | value: 53.931 |
| 384 | - type: ndcg_at_1 |
| 385 | value: 44.513999999999996 |
| 386 | - type: ndcg_at_10 |
| 387 | value: 57.67400000000001 |
| 388 | - type: ndcg_at_100 |
| 389 | value: 61.663999999999994 |
| 390 | - type: ndcg_at_1000 |
| 391 | value: 62.743 |
| 392 | - type: ndcg_at_3 |
| 393 | value: 51.964 |
| 394 | - type: ndcg_at_5 |
| 395 | value: 54.773 |
| 396 | - type: precision_at_1 |
| 397 | value: 44.513999999999996 |
| 398 | - type: precision_at_10 |
| 399 | value: 9.423 |
| 400 | - type: precision_at_100 |
| 401 | value: 1.2309999999999999 |
| 402 | - type: precision_at_1000 |
| 403 | value: 0.13699999999999998 |
| 404 | - type: precision_at_3 |
| 405 | value: 23.323 |
| 406 | - type: precision_at_5 |
| 407 | value: 16.163 |
| 408 | - type: recall_at_1 |
| 409 | value: 38.83 |
| 410 | - type: recall_at_10 |
| 411 | value: 72.327 |
| 412 | - type: recall_at_100 |
| 413 | value: 89.519 |
| 414 | - type: recall_at_1000 |
| 415 | value: 97.041 |
| 416 | - type: recall_at_3 |
| 417 | value: 57.206 |
| 418 | - type: recall_at_5 |
| 419 | value: 63.88399999999999 |
| 420 | - task: |
| 421 | type: Retrieval |
| 422 | dataset: |
| 423 | type: BeIR/cqadupstack |
| 424 | name: MTEB CQADupstackGisRetrieval |
| 425 | config: default |
| 426 | split: test |
| 427 | revision: None |
| 428 | metrics: |
| 429 | - type: map_at_1 |
| 430 | value: 25.484 |
| 431 | - type: map_at_10 |
| 432 | value: 34.527 |
| 433 | - type: map_at_100 |
| 434 | value: 35.661 |
| 435 | - type: map_at_1000 |
| 436 | value: 35.739 |
| 437 | - type: map_at_3 |
| 438 | value: 32.199 |
| 439 | - type: map_at_5 |
| 440 | value: 33.632 |
| 441 | - type: mrr_at_1 |
| 442 | value: 27.458 |
| 443 | - type: mrr_at_10 |
| 444 | value: 36.543 |
| 445 | - type: mrr_at_100 |
| 446 | value: 37.482 |
| 447 | - type: mrr_at_1000 |
| 448 | value: 37.543 |
| 449 | - type: mrr_at_3 |
| 450 | value: 34.256 |
| 451 | - type: mrr_at_5 |
| 452 | value: 35.618 |
| 453 | - type: ndcg_at_1 |
| 454 | value: 27.458 |
| 455 | - type: ndcg_at_10 |
| 456 | value: 39.396 |
| 457 | - type: ndcg_at_100 |
| 458 | value: 44.742 |
| 459 | - type: ndcg_at_1000 |
| 460 | value: 46.708 |
| 461 | - type: ndcg_at_3 |
| 462 | value: 34.817 |
| 463 | - type: ndcg_at_5 |
| 464 | value: 37.247 |
| 465 | - type: precision_at_1 |
| 466 | value: 27.458 |
| 467 | - type: precision_at_10 |
| 468 | value: 5.976999999999999 |
| 469 | - type: precision_at_100 |
| 470 | value: 0.907 |
| 471 | - type: precision_at_1000 |
| 472 | value: 0.11100000000000002 |
| 473 | - type: precision_at_3 |
| 474 | value: 14.878 |
| 475 | - type: precision_at_5 |
| 476 | value: 10.35 |
| 477 | - type: recall_at_1 |
| 478 | value: 25.484 |
| 479 | - type: recall_at_10 |
| 480 | value: 52.317 |
| 481 | - type: recall_at_100 |
| 482 | value: 76.701 |
| 483 | - type: recall_at_1000 |
| 484 | value: 91.408 |
| 485 | - type: recall_at_3 |
| 486 | value: 40.043 |
| 487 | - type: recall_at_5 |
| 488 | value: 45.879 |
| 489 | - task: |
| 490 | type: Retrieval |
| 491 | dataset: |
| 492 | type: BeIR/cqadupstack |
| 493 | name: MTEB CQADupstackMathematicaRetrieval |
| 494 | config: default |
| 495 | split: test |
| 496 | revision: None |
| 497 | metrics: |
| 498 | - type: map_at_1 |
| 499 | value: 16.719 |
| 500 | - type: map_at_10 |
| 501 | value: 25.269000000000002 |
| 502 | - type: map_at_100 |
| 503 | value: 26.442 |
| 504 | - type: map_at_1000 |
| 505 | value: 26.557 |
| 506 | - type: map_at_3 |
| 507 | value: 22.56 |
| 508 | - type: map_at_5 |
| 509 | value: 24.082 |
| 510 | - type: mrr_at_1 |
| 511 | value: 20.896 |
| 512 | - type: mrr_at_10 |
| 513 | value: 29.982999999999997 |
| 514 | - type: mrr_at_100 |
| 515 | value: 30.895 |
| 516 | - type: mrr_at_1000 |
| 517 | value: 30.961 |
| 518 | - type: mrr_at_3 |
| 519 | value: 27.239 |
| 520 | - type: mrr_at_5 |
| 521 | value: 28.787000000000003 |
| 522 | - type: ndcg_at_1 |
| 523 | value: 20.896 |
| 524 | - type: ndcg_at_10 |
| 525 | value: 30.814000000000004 |
| 526 | - type: ndcg_at_100 |
| 527 | value: 36.418 |
| 528 | - type: ndcg_at_1000 |
| 529 | value: 39.182 |
| 530 | - type: ndcg_at_3 |
| 531 | value: 25.807999999999996 |
| 532 | - type: ndcg_at_5 |
| 533 | value: 28.143 |
| 534 | - type: precision_at_1 |
| 535 | value: 20.896 |
| 536 | - type: precision_at_10 |
| 537 | value: 5.821 |
| 538 | - type: precision_at_100 |
| 539 | value: 0.991 |
| 540 | - type: precision_at_1000 |
| 541 | value: 0.136 |
| 542 | - type: precision_at_3 |
| 543 | value: 12.562000000000001 |
| 544 | - type: precision_at_5 |
| 545 | value: 9.254 |
| 546 | - type: recall_at_1 |
| 547 | value: 16.719 |
| 548 | - type: recall_at_10 |
| 549 | value: 43.155 |
| 550 | - type: recall_at_100 |
| 551 | value: 67.831 |
| 552 | - type: recall_at_1000 |
| 553 | value: 87.617 |
| 554 | - type: recall_at_3 |
| 555 | value: 29.259 |
| 556 | - type: recall_at_5 |
| 557 | value: 35.260999999999996 |
| 558 | - task: |
| 559 | type: Retrieval |
| 560 | dataset: |
| 561 | type: BeIR/cqadupstack |
| 562 | name: MTEB CQADupstackPhysicsRetrieval |
| 563 | config: default |
| 564 | split: test |
| 565 | revision: None |
| 566 | metrics: |
| 567 | - type: map_at_1 |
| 568 | value: 29.398999999999997 |
| 569 | - type: map_at_10 |
| 570 | value: 39.876 |
| 571 | - type: map_at_100 |
| 572 | value: 41.205999999999996 |
| 573 | - type: map_at_1000 |
| 574 | value: 41.321999999999996 |
| 575 | - type: map_at_3 |
| 576 | value: 36.588 |
| 577 | - type: map_at_5 |
| 578 | value: 38.538 |
| 579 | - type: mrr_at_1 |
| 580 | value: 35.9 |
| 581 | - type: mrr_at_10 |
| 582 | value: 45.528 |
| 583 | - type: mrr_at_100 |
| 584 | value: 46.343 |
| 585 | - type: mrr_at_1000 |
| 586 | value: 46.388 |
| 587 | - type: mrr_at_3 |
| 588 | value: 42.862 |
| 589 | - type: mrr_at_5 |
| 590 | value: 44.440000000000005 |
| 591 | - type: ndcg_at_1 |
| 592 | value: 35.9 |
| 593 | - type: ndcg_at_10 |
| 594 | value: 45.987 |
| 595 | - type: ndcg_at_100 |
| 596 | value: 51.370000000000005 |
| 597 | - type: ndcg_at_1000 |
| 598 | value: 53.400000000000006 |
| 599 | - type: ndcg_at_3 |
| 600 | value: 40.841 |
| 601 | - type: ndcg_at_5 |
| 602 | value: 43.447 |
| 603 | - type: precision_at_1 |
| 604 | value: 35.9 |
| 605 | - type: precision_at_10 |
| 606 | value: 8.393 |
| 607 | - type: precision_at_100 |
| 608 | value: 1.283 |
| 609 | - type: precision_at_1000 |
| 610 | value: 0.166 |
| 611 | - type: precision_at_3 |
| 612 | value: 19.538 |
| 613 | - type: precision_at_5 |
| 614 | value: 13.975000000000001 |
| 615 | - type: recall_at_1 |
| 616 | value: 29.398999999999997 |
| 617 | - type: recall_at_10 |
| 618 | value: 58.361 |
| 619 | - type: recall_at_100 |
| 620 | value: 81.081 |
| 621 | - type: recall_at_1000 |
| 622 | value: 94.004 |
| 623 | - type: recall_at_3 |
| 624 | value: 43.657000000000004 |
| 625 | - type: recall_at_5 |
| 626 | value: 50.519999999999996 |
| 627 | - task: |
| 628 | type: Retrieval |
| 629 | dataset: |
| 630 | type: BeIR/cqadupstack |
| 631 | name: MTEB CQADupstackProgrammersRetrieval |
| 632 | config: default |
| 633 | split: test |
| 634 | revision: None |
| 635 | metrics: |
| 636 | - type: map_at_1 |
| 637 | value: 21.589 |
| 638 | - type: map_at_10 |
| 639 | value: 31.608999999999998 |
| 640 | - type: map_at_100 |
| 641 | value: 33.128 |
| 642 | - type: map_at_1000 |
| 643 | value: 33.247 |
| 644 | - type: map_at_3 |
| 645 | value: 28.671999999999997 |
| 646 | - type: map_at_5 |
| 647 | value: 30.233999999999998 |
| 648 | - type: mrr_at_1 |
| 649 | value: 26.712000000000003 |
| 650 | - type: mrr_at_10 |
| 651 | value: 36.713 |
| 652 | - type: mrr_at_100 |
| 653 | value: 37.713 |
| 654 | - type: mrr_at_1000 |
| 655 | value: 37.771 |
| 656 | - type: mrr_at_3 |
| 657 | value: 34.075 |
| 658 | - type: mrr_at_5 |
| 659 | value: 35.451 |
| 660 | - type: ndcg_at_1 |
| 661 | value: 26.712000000000003 |
| 662 | - type: ndcg_at_10 |
| 663 | value: 37.519999999999996 |
| 664 | - type: ndcg_at_100 |
| 665 | value: 43.946000000000005 |
| 666 | - type: ndcg_at_1000 |
| 667 | value: 46.297 |
| 668 | - type: ndcg_at_3 |
| 669 | value: 32.551 |
| 670 | - type: ndcg_at_5 |
| 671 | value: 34.660999999999994 |
| 672 | - type: precision_at_1 |
| 673 | value: 26.712000000000003 |
| 674 | - type: precision_at_10 |
| 675 | value: 7.066 |
| 676 | - type: precision_at_100 |
| 677 | value: 1.216 |
| 678 | - type: precision_at_1000 |
| 679 | value: 0.157 |
| 680 | - type: precision_at_3 |
| 681 | value: 15.906 |
| 682 | - type: precision_at_5 |
| 683 | value: 11.437999999999999 |
| 684 | - type: recall_at_1 |
| 685 | value: 21.589 |
| 686 | - type: recall_at_10 |
| 687 | value: 50.090999999999994 |
| 688 | - type: recall_at_100 |
| 689 | value: 77.43900000000001 |
| 690 | - type: recall_at_1000 |
| 691 | value: 93.35900000000001 |
| 692 | - type: recall_at_3 |
| 693 | value: 36.028999999999996 |
| 694 | - type: recall_at_5 |
| 695 | value: 41.698 |
| 696 | - task: |
| 697 | type: Retrieval |
| 698 | dataset: |
| 699 | type: BeIR/cqadupstack |
| 700 | name: MTEB CQADupstackRetrieval |
| 701 | config: default |
| 702 | split: test |
| 703 | revision: None |
| 704 | metrics: |
| 705 | - type: map_at_1 |
| 706 | value: 25.121666666666663 |
| 707 | - type: map_at_10 |
| 708 | value: 34.46258333333334 |
| 709 | - type: map_at_100 |
| 710 | value: 35.710499999999996 |
| 711 | - type: map_at_1000 |
| 712 | value: 35.82691666666666 |
| 713 | - type: map_at_3 |
| 714 | value: 31.563249999999996 |
| 715 | - type: map_at_5 |
| 716 | value: 33.189750000000004 |
| 717 | - type: mrr_at_1 |
| 718 | value: 29.66441666666667 |
| 719 | - type: mrr_at_10 |
| 720 | value: 38.5455 |
| 721 | - type: mrr_at_100 |
| 722 | value: 39.39566666666667 |
| 723 | - type: mrr_at_1000 |
| 724 | value: 39.45325 |
| 725 | - type: mrr_at_3 |
| 726 | value: 36.003333333333345 |
| 727 | - type: mrr_at_5 |
| 728 | value: 37.440916666666666 |
| 729 | - type: ndcg_at_1 |
| 730 | value: 29.66441666666667 |
| 731 | - type: ndcg_at_10 |
| 732 | value: 39.978416666666675 |
| 733 | - type: ndcg_at_100 |
| 734 | value: 45.278666666666666 |
| 735 | - type: ndcg_at_1000 |
| 736 | value: 47.52275 |
| 737 | - type: ndcg_at_3 |
| 738 | value: 35.00058333333334 |
| 739 | - type: ndcg_at_5 |
| 740 | value: 37.34908333333333 |
| 741 | - type: precision_at_1 |
| 742 | value: 29.66441666666667 |
| 743 | - type: precision_at_10 |
| 744 | value: 7.094500000000001 |
| 745 | - type: precision_at_100 |
| 746 | value: 1.1523333333333332 |
| 747 | - type: precision_at_1000 |
| 748 | value: 0.15358333333333332 |
| 749 | - type: precision_at_3 |
| 750 | value: 16.184166666666663 |
| 751 | - type: precision_at_5 |
| 752 | value: 11.6005 |
| 753 | - type: recall_at_1 |
| 754 | value: 25.121666666666663 |
| 755 | - type: recall_at_10 |
| 756 | value: 52.23975000000001 |
| 757 | - type: recall_at_100 |
| 758 | value: 75.48408333333333 |
| 759 | - type: recall_at_1000 |
| 760 | value: 90.95316666666668 |
| 761 | - type: recall_at_3 |
| 762 | value: 38.38458333333333 |
| 763 | - type: recall_at_5 |
| 764 | value: 44.39933333333333 |
| 765 | - task: |
| 766 | type: Retrieval |
| 767 | dataset: |
| 768 | type: BeIR/cqadupstack |
| 769 | name: MTEB CQADupstackStatsRetrieval |
| 770 | config: default |
| 771 | split: test |
| 772 | revision: None |
| 773 | metrics: |
| 774 | - type: map_at_1 |
| 775 | value: 23.569000000000003 |
| 776 | - type: map_at_10 |
| 777 | value: 30.389 |
| 778 | - type: map_at_100 |
| 779 | value: 31.396 |
| 780 | - type: map_at_1000 |
| 781 | value: 31.493 |
| 782 | - type: map_at_3 |
| 783 | value: 28.276 |
| 784 | - type: map_at_5 |
| 785 | value: 29.459000000000003 |
| 786 | - type: mrr_at_1 |
| 787 | value: 26.534000000000002 |
| 788 | - type: mrr_at_10 |
| 789 | value: 33.217999999999996 |
| 790 | - type: mrr_at_100 |
| 791 | value: 34.054 |
| 792 | - type: mrr_at_1000 |
| 793 | value: 34.12 |
| 794 | - type: mrr_at_3 |
| 795 | value: 31.058000000000003 |
| 796 | - type: mrr_at_5 |
| 797 | value: 32.330999999999996 |
| 798 | - type: ndcg_at_1 |
| 799 | value: 26.534000000000002 |
| 800 | - type: ndcg_at_10 |
| 801 | value: 34.608 |
| 802 | - type: ndcg_at_100 |
| 803 | value: 39.391999999999996 |
| 804 | - type: ndcg_at_1000 |
| 805 | value: 41.837999999999994 |
| 806 | - type: ndcg_at_3 |
| 807 | value: 30.564999999999998 |
| 808 | - type: ndcg_at_5 |
| 809 | value: 32.509 |
| 810 | - type: precision_at_1 |
| 811 | value: 26.534000000000002 |
| 812 | - type: precision_at_10 |
| 813 | value: 5.414 |
| 814 | - type: precision_at_100 |
| 815 | value: 0.847 |
| 816 | - type: precision_at_1000 |
| 817 | value: 0.11399999999999999 |
| 818 | - type: precision_at_3 |
| 819 | value: 12.986 |
| 820 | - type: precision_at_5 |
| 821 | value: 9.202 |
| 822 | - type: recall_at_1 |
| 823 | value: 23.569000000000003 |
| 824 | - type: recall_at_10 |
| 825 | value: 44.896 |
| 826 | - type: recall_at_100 |
| 827 | value: 66.476 |
| 828 | - type: recall_at_1000 |
| 829 | value: 84.548 |
| 830 | - type: recall_at_3 |
| 831 | value: 33.79 |
| 832 | - type: recall_at_5 |
| 833 | value: 38.512 |
| 834 | - task: |
| 835 | type: Retrieval |
| 836 | dataset: |
| 837 | type: BeIR/cqadupstack |
| 838 | name: MTEB CQADupstackTexRetrieval |
| 839 | config: default |
| 840 | split: test |
| 841 | revision: None |
| 842 | metrics: |
| 843 | - type: map_at_1 |
| 844 | value: 16.36 |
| 845 | - type: map_at_10 |
| 846 | value: 23.57 |
| 847 | - type: map_at_100 |
| 848 | value: 24.698999999999998 |
| 849 | - type: map_at_1000 |
| 850 | value: 24.834999999999997 |
| 851 | - type: map_at_3 |
| 852 | value: 21.093 |
| 853 | - type: map_at_5 |
| 854 | value: 22.418 |
| 855 | - type: mrr_at_1 |
| 856 | value: 19.718 |
| 857 | - type: mrr_at_10 |
| 858 | value: 27.139999999999997 |
| 859 | - type: mrr_at_100 |
| 860 | value: 28.097 |
| 861 | - type: mrr_at_1000 |
| 862 | value: 28.177999999999997 |
| 863 | - type: mrr_at_3 |
| 864 | value: 24.805 |
| 865 | - type: mrr_at_5 |
| 866 | value: 26.121 |
| 867 | - type: ndcg_at_1 |
| 868 | value: 19.718 |
| 869 | - type: ndcg_at_10 |
| 870 | value: 28.238999999999997 |
| 871 | - type: ndcg_at_100 |
| 872 | value: 33.663 |
| 873 | - type: ndcg_at_1000 |
| 874 | value: 36.763 |
| 875 | - type: ndcg_at_3 |
| 876 | value: 23.747 |
| 877 | - type: ndcg_at_5 |
| 878 | value: 25.796000000000003 |
| 879 | - type: precision_at_1 |
| 880 | value: 19.718 |
| 881 | - type: precision_at_10 |
| 882 | value: 5.282 |
| 883 | - type: precision_at_100 |
| 884 | value: 0.9390000000000001 |
| 885 | - type: precision_at_1000 |
| 886 | value: 0.13899999999999998 |
| 887 | - type: precision_at_3 |
| 888 | value: 11.264000000000001 |
| 889 | - type: precision_at_5 |
| 890 | value: 8.341 |
| 891 | - type: recall_at_1 |
| 892 | value: 16.36 |
| 893 | - type: recall_at_10 |
| 894 | value: 38.669 |
| 895 | - type: recall_at_100 |
| 896 | value: 63.184 |
| 897 | - type: recall_at_1000 |
| 898 | value: 85.33800000000001 |
| 899 | - type: recall_at_3 |
| 900 | value: 26.214 |
| 901 | - type: recall_at_5 |
| 902 | value: 31.423000000000002 |
| 903 | - task: |
| 904 | type: Retrieval |
| 905 | dataset: |
| 906 | type: BeIR/cqadupstack |
| 907 | name: MTEB CQADupstackUnixRetrieval |
| 908 | config: default |
| 909 | split: test |
| 910 | revision: None |
| 911 | metrics: |
| 912 | - type: map_at_1 |
| 913 | value: 25.618999999999996 |
| 914 | - type: map_at_10 |
| 915 | value: 34.361999999999995 |
| 916 | - type: map_at_100 |
| 917 | value: 35.534 |
| 918 | - type: map_at_1000 |
| 919 | value: 35.634 |
| 920 | - type: map_at_3 |
| 921 | value: 31.402 |
| 922 | - type: map_at_5 |
| 923 | value: 32.815 |
| 924 | - type: mrr_at_1 |
| 925 | value: 30.037000000000003 |
| 926 | - type: mrr_at_10 |
| 927 | value: 38.284 |
| 928 | - type: mrr_at_100 |
| 929 | value: 39.141999999999996 |
| 930 | - type: mrr_at_1000 |
| 931 | value: 39.2 |
| 932 | - type: mrr_at_3 |
| 933 | value: 35.603 |
| 934 | - type: mrr_at_5 |
| 935 | value: 36.867 |
| 936 | - type: ndcg_at_1 |
| 937 | value: 30.037000000000003 |
| 938 | - type: ndcg_at_10 |
| 939 | value: 39.87 |
| 940 | - type: ndcg_at_100 |
| 941 | value: 45.243 |
| 942 | - type: ndcg_at_1000 |
| 943 | value: 47.507 |
| 944 | - type: ndcg_at_3 |
| 945 | value: 34.371 |
| 946 | - type: ndcg_at_5 |
| 947 | value: 36.521 |
| 948 | - type: precision_at_1 |
| 949 | value: 30.037000000000003 |
| 950 | - type: precision_at_10 |
| 951 | value: 6.819 |
| 952 | - type: precision_at_100 |
| 953 | value: 1.0699999999999998 |
| 954 | - type: precision_at_1000 |
| 955 | value: 0.13699999999999998 |
| 956 | - type: precision_at_3 |
| 957 | value: 15.392 |
| 958 | - type: precision_at_5 |
| 959 | value: 10.821 |
| 960 | - type: recall_at_1 |
| 961 | value: 25.618999999999996 |
| 962 | - type: recall_at_10 |
| 963 | value: 52.869 |
| 964 | - type: recall_at_100 |
| 965 | value: 76.395 |
| 966 | - type: recall_at_1000 |
| 967 | value: 92.19500000000001 |
| 968 | - type: recall_at_3 |
| 969 | value: 37.943 |
| 970 | - type: recall_at_5 |
| 971 | value: 43.342999999999996 |
| 972 | - task: |
| 973 | type: Retrieval |
| 974 | dataset: |
| 975 | type: BeIR/cqadupstack |
| 976 | name: MTEB CQADupstackWebmastersRetrieval |
| 977 | config: default |
| 978 | split: test |
| 979 | revision: None |
| 980 | metrics: |
| 981 | - type: map_at_1 |
| 982 | value: 23.283 |
| 983 | - type: map_at_10 |
| 984 | value: 32.155 |
| 985 | - type: map_at_100 |
| 986 | value: 33.724 |
| 987 | - type: map_at_1000 |
| 988 | value: 33.939 |
| 989 | - type: map_at_3 |
| 990 | value: 29.018 |
| 991 | - type: map_at_5 |
| 992 | value: 30.864000000000004 |
| 993 | - type: mrr_at_1 |
| 994 | value: 28.063 |
| 995 | - type: mrr_at_10 |
| 996 | value: 36.632 |
| 997 | - type: mrr_at_100 |
| 998 | value: 37.606 |
| 999 | - type: mrr_at_1000 |
| 1000 | value: 37.671 |
| 1001 | - type: mrr_at_3 |
| 1002 | value: 33.992 |
| 1003 | - type: mrr_at_5 |
| 1004 | value: 35.613 |
| 1005 | - type: ndcg_at_1 |
| 1006 | value: 28.063 |
| 1007 | - type: ndcg_at_10 |
| 1008 | value: 38.024 |
| 1009 | - type: ndcg_at_100 |
| 1010 | value: 44.292 |
| 1011 | - type: ndcg_at_1000 |
| 1012 | value: 46.818 |
| 1013 | - type: ndcg_at_3 |
| 1014 | value: 32.965 |
| 1015 | - type: ndcg_at_5 |
| 1016 | value: 35.562 |
| 1017 | - type: precision_at_1 |
| 1018 | value: 28.063 |
| 1019 | - type: precision_at_10 |
| 1020 | value: 7.352 |
| 1021 | - type: precision_at_100 |
| 1022 | value: 1.514 |
| 1023 | - type: precision_at_1000 |
| 1024 | value: 0.23800000000000002 |
| 1025 | - type: precision_at_3 |
| 1026 | value: 15.481 |
| 1027 | - type: precision_at_5 |
| 1028 | value: 11.542 |
| 1029 | - type: recall_at_1 |
| 1030 | value: 23.283 |
| 1031 | - type: recall_at_10 |
| 1032 | value: 49.756 |
| 1033 | - type: recall_at_100 |
| 1034 | value: 78.05 |
| 1035 | - type: recall_at_1000 |
| 1036 | value: 93.854 |
| 1037 | - type: recall_at_3 |
| 1038 | value: 35.408 |
| 1039 | - type: recall_at_5 |
| 1040 | value: 42.187000000000005 |
| 1041 | - task: |
| 1042 | type: Retrieval |
| 1043 | dataset: |
| 1044 | type: BeIR/cqadupstack |
| 1045 | name: MTEB CQADupstackWordpressRetrieval |
| 1046 | config: default |
| 1047 | split: test |
| 1048 | revision: None |
| 1049 | metrics: |
| 1050 | - type: map_at_1 |
| 1051 | value: 19.201999999999998 |
| 1052 | - type: map_at_10 |
| 1053 | value: 26.826 |
| 1054 | - type: map_at_100 |
| 1055 | value: 27.961000000000002 |
| 1056 | - type: map_at_1000 |
| 1057 | value: 28.066999999999997 |
| 1058 | - type: map_at_3 |
| 1059 | value: 24.237000000000002 |
| 1060 | - type: map_at_5 |
| 1061 | value: 25.811 |
| 1062 | - type: mrr_at_1 |
| 1063 | value: 20.887 |
| 1064 | - type: mrr_at_10 |
| 1065 | value: 28.660000000000004 |
| 1066 | - type: mrr_at_100 |
| 1067 | value: 29.660999999999998 |
| 1068 | - type: mrr_at_1000 |
| 1069 | value: 29.731 |
| 1070 | - type: mrr_at_3 |
| 1071 | value: 26.155 |
| 1072 | - type: mrr_at_5 |
| 1073 | value: 27.68 |
| 1074 | - type: ndcg_at_1 |
| 1075 | value: 20.887 |
| 1076 | - type: ndcg_at_10 |
| 1077 | value: 31.523 |
| 1078 | - type: ndcg_at_100 |
| 1079 | value: 37.055 |
| 1080 | - type: ndcg_at_1000 |
| 1081 | value: 39.579 |
| 1082 | - type: ndcg_at_3 |
| 1083 | value: 26.529000000000003 |
| 1084 | - type: ndcg_at_5 |
| 1085 | value: 29.137 |
| 1086 | - type: precision_at_1 |
| 1087 | value: 20.887 |
| 1088 | - type: precision_at_10 |
| 1089 | value: 5.065 |
| 1090 | - type: precision_at_100 |
| 1091 | value: 0.856 |
| 1092 | - type: precision_at_1000 |
| 1093 | value: 0.11900000000000001 |
| 1094 | - type: precision_at_3 |
| 1095 | value: 11.399 |
| 1096 | - type: precision_at_5 |
| 1097 | value: 8.392 |
| 1098 | - type: recall_at_1 |
| 1099 | value: 19.201999999999998 |
| 1100 | - type: recall_at_10 |
| 1101 | value: 44.285000000000004 |
| 1102 | - type: recall_at_100 |
| 1103 | value: 69.768 |
| 1104 | - type: recall_at_1000 |
| 1105 | value: 88.302 |
| 1106 | - type: recall_at_3 |
| 1107 | value: 30.804 |
| 1108 | - type: recall_at_5 |
| 1109 | value: 37.039 |
| 1110 | - task: |
| 1111 | type: Retrieval |
| 1112 | dataset: |
| 1113 | type: climate-fever |
| 1114 | name: MTEB ClimateFEVER |
| 1115 | config: default |
| 1116 | split: test |
| 1117 | revision: None |
| 1118 | metrics: |
| 1119 | - type: map_at_1 |
| 1120 | value: 11.244 |
| 1121 | - type: map_at_10 |
| 1122 | value: 18.956 |
| 1123 | - type: map_at_100 |
| 1124 | value: 20.674 |
| 1125 | - type: map_at_1000 |
| 1126 | value: 20.863 |
| 1127 | - type: map_at_3 |
| 1128 | value: 15.923000000000002 |
| 1129 | - type: map_at_5 |
| 1130 | value: 17.518 |
| 1131 | - type: mrr_at_1 |
| 1132 | value: 25.080999999999996 |
| 1133 | - type: mrr_at_10 |
| 1134 | value: 35.94 |
| 1135 | - type: mrr_at_100 |
| 1136 | value: 36.969 |
| 1137 | - type: mrr_at_1000 |
| 1138 | value: 37.013 |
| 1139 | - type: mrr_at_3 |
| 1140 | value: 32.617000000000004 |
| 1141 | - type: mrr_at_5 |
| 1142 | value: 34.682 |
| 1143 | - type: ndcg_at_1 |
| 1144 | value: 25.080999999999996 |
| 1145 | - type: ndcg_at_10 |
| 1146 | value: 26.539 |
| 1147 | - type: ndcg_at_100 |
| 1148 | value: 33.601 |
| 1149 | - type: ndcg_at_1000 |
| 1150 | value: 37.203 |
| 1151 | - type: ndcg_at_3 |
| 1152 | value: 21.695999999999998 |
| 1153 | - type: ndcg_at_5 |
| 1154 | value: 23.567 |
| 1155 | - type: precision_at_1 |
| 1156 | value: 25.080999999999996 |
| 1157 | - type: precision_at_10 |
| 1158 | value: 8.143 |
| 1159 | - type: precision_at_100 |
| 1160 | value: 1.5650000000000002 |
| 1161 | - type: precision_at_1000 |
| 1162 | value: 0.22300000000000003 |
| 1163 | - type: precision_at_3 |
| 1164 | value: 15.983 |
| 1165 | - type: precision_at_5 |
| 1166 | value: 12.417 |
| 1167 | - type: recall_at_1 |
| 1168 | value: 11.244 |
| 1169 | - type: recall_at_10 |
| 1170 | value: 31.457 |
| 1171 | - type: recall_at_100 |
| 1172 | value: 55.92 |
| 1173 | - type: recall_at_1000 |
| 1174 | value: 76.372 |
| 1175 | - type: recall_at_3 |
| 1176 | value: 19.784 |
| 1177 | - type: recall_at_5 |
| 1178 | value: 24.857000000000003 |
| 1179 | - task: |
| 1180 | type: Retrieval |
| 1181 | dataset: |
| 1182 | type: dbpedia-entity |
| 1183 | name: MTEB DBPedia |
| 1184 | config: default |
| 1185 | split: test |
| 1186 | revision: None |
| 1187 | metrics: |
| 1188 | - type: map_at_1 |
| 1189 | value: 8.595 |
| 1190 | - type: map_at_10 |
| 1191 | value: 18.75 |
| 1192 | - type: map_at_100 |
| 1193 | value: 26.354 |
| 1194 | - type: map_at_1000 |
| 1195 | value: 27.912 |
| 1196 | - type: map_at_3 |
| 1197 | value: 13.794 |
| 1198 | - type: map_at_5 |
| 1199 | value: 16.021 |
| 1200 | - type: mrr_at_1 |
| 1201 | value: 65.75 |
| 1202 | - type: mrr_at_10 |
| 1203 | value: 73.837 |
| 1204 | - type: mrr_at_100 |
| 1205 | value: 74.22800000000001 |
| 1206 | - type: mrr_at_1000 |
| 1207 | value: 74.234 |
| 1208 | - type: mrr_at_3 |
| 1209 | value: 72.5 |
| 1210 | - type: mrr_at_5 |
| 1211 | value: 73.387 |
| 1212 | - type: ndcg_at_1 |
| 1213 | value: 52.625 |
| 1214 | - type: ndcg_at_10 |
| 1215 | value: 39.101 |
| 1216 | - type: ndcg_at_100 |
| 1217 | value: 43.836000000000006 |
| 1218 | - type: ndcg_at_1000 |
| 1219 | value: 51.086 |
| 1220 | - type: ndcg_at_3 |
| 1221 | value: 44.229 |
| 1222 | - type: ndcg_at_5 |
| 1223 | value: 41.555 |
| 1224 | - type: precision_at_1 |
| 1225 | value: 65.75 |
| 1226 | - type: precision_at_10 |
| 1227 | value: 30.45 |
| 1228 | - type: precision_at_100 |
| 1229 | value: 9.81 |
| 1230 | - type: precision_at_1000 |
| 1231 | value: 2.045 |
| 1232 | - type: precision_at_3 |
| 1233 | value: 48.667 |
| 1234 | - type: precision_at_5 |
| 1235 | value: 40.8 |
| 1236 | - type: recall_at_1 |
| 1237 | value: 8.595 |
| 1238 | - type: recall_at_10 |
| 1239 | value: 24.201 |
| 1240 | - type: recall_at_100 |
| 1241 | value: 50.096 |
| 1242 | - type: recall_at_1000 |
| 1243 | value: 72.677 |
| 1244 | - type: recall_at_3 |
| 1245 | value: 15.212 |
| 1246 | - type: recall_at_5 |
| 1247 | value: 18.745 |
| 1248 | - task: |
| 1249 | type: Classification |
| 1250 | dataset: |
| 1251 | type: mteb/emotion |
| 1252 | name: MTEB EmotionClassification |
| 1253 | config: default |
| 1254 | split: test |
| 1255 | revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
| 1256 | metrics: |
| 1257 | - type: accuracy |
| 1258 | value: 46.565 |
| 1259 | - type: f1 |
| 1260 | value: 41.49914329345582 |
| 1261 | - task: |
| 1262 | type: Retrieval |
| 1263 | dataset: |
| 1264 | type: fever |
| 1265 | name: MTEB FEVER |
| 1266 | config: default |
| 1267 | split: test |
| 1268 | revision: None |
| 1269 | metrics: |
| 1270 | - type: map_at_1 |
| 1271 | value: 66.60000000000001 |
| 1272 | - type: map_at_10 |
| 1273 | value: 76.838 |
| 1274 | - type: map_at_100 |
| 1275 | value: 77.076 |
| 1276 | - type: map_at_1000 |
| 1277 | value: 77.09 |
| 1278 | - type: map_at_3 |
| 1279 | value: 75.545 |
| 1280 | - type: map_at_5 |
| 1281 | value: 76.39 |
| 1282 | - type: mrr_at_1 |
| 1283 | value: 71.707 |
| 1284 | - type: mrr_at_10 |
| 1285 | value: 81.514 |
| 1286 | - type: mrr_at_100 |
| 1287 | value: 81.64099999999999 |
| 1288 | - type: mrr_at_1000 |
| 1289 | value: 81.645 |
| 1290 | - type: mrr_at_3 |
| 1291 | value: 80.428 |
| 1292 | - type: mrr_at_5 |
| 1293 | value: 81.159 |
| 1294 | - type: ndcg_at_1 |
| 1295 | value: 71.707 |
| 1296 | - type: ndcg_at_10 |
| 1297 | value: 81.545 |
| 1298 | - type: ndcg_at_100 |
| 1299 | value: 82.477 |
| 1300 | - type: ndcg_at_1000 |
| 1301 | value: 82.73899999999999 |
| 1302 | - type: ndcg_at_3 |
| 1303 | value: 79.292 |
| 1304 | - type: ndcg_at_5 |
| 1305 | value: 80.599 |
| 1306 | - type: precision_at_1 |
| 1307 | value: 71.707 |
| 1308 | - type: precision_at_10 |
| 1309 | value: 10.035 |
| 1310 | - type: precision_at_100 |
| 1311 | value: 1.068 |
| 1312 | - type: precision_at_1000 |
| 1313 | value: 0.11100000000000002 |
| 1314 | - type: precision_at_3 |
| 1315 | value: 30.918 |
| 1316 | - type: precision_at_5 |
| 1317 | value: 19.328 |
| 1318 | - type: recall_at_1 |
| 1319 | value: 66.60000000000001 |
| 1320 | - type: recall_at_10 |
| 1321 | value: 91.353 |
| 1322 | - type: recall_at_100 |
| 1323 | value: 95.21 |
| 1324 | - type: recall_at_1000 |
| 1325 | value: 96.89999999999999 |
| 1326 | - type: recall_at_3 |
| 1327 | value: 85.188 |
| 1328 | - type: recall_at_5 |
| 1329 | value: 88.52 |
| 1330 | - task: |
| 1331 | type: Retrieval |
| 1332 | dataset: |
| 1333 | type: fiqa |
| 1334 | name: MTEB FiQA2018 |
| 1335 | config: default |
| 1336 | split: test |
| 1337 | revision: None |
| 1338 | metrics: |
| 1339 | - type: map_at_1 |
| 1340 | value: 19.338 |
| 1341 | - type: map_at_10 |
| 1342 | value: 31.752000000000002 |
| 1343 | - type: map_at_100 |
| 1344 | value: 33.516 |
| 1345 | - type: map_at_1000 |
| 1346 | value: 33.694 |
| 1347 | - type: map_at_3 |
| 1348 | value: 27.716 |
| 1349 | - type: map_at_5 |
| 1350 | value: 29.67 |
| 1351 | - type: mrr_at_1 |
| 1352 | value: 38.117000000000004 |
| 1353 | - type: mrr_at_10 |
| 1354 | value: 47.323 |
| 1355 | - type: mrr_at_100 |
| 1356 | value: 48.13 |
| 1357 | - type: mrr_at_1000 |
| 1358 | value: 48.161 |
| 1359 | - type: mrr_at_3 |
| 1360 | value: 45.062000000000005 |
| 1361 | - type: mrr_at_5 |
| 1362 | value: 46.358 |
| 1363 | - type: ndcg_at_1 |
| 1364 | value: 38.117000000000004 |
| 1365 | - type: ndcg_at_10 |
| 1366 | value: 39.353 |
| 1367 | - type: ndcg_at_100 |
| 1368 | value: 46.044000000000004 |
| 1369 | - type: ndcg_at_1000 |
| 1370 | value: 49.083 |
| 1371 | - type: ndcg_at_3 |
| 1372 | value: 35.891 |
| 1373 | - type: ndcg_at_5 |
| 1374 | value: 36.661 |
| 1375 | - type: precision_at_1 |
| 1376 | value: 38.117000000000004 |
| 1377 | - type: precision_at_10 |
| 1378 | value: 11.187999999999999 |
| 1379 | - type: precision_at_100 |
| 1380 | value: 1.802 |
| 1381 | - type: precision_at_1000 |
| 1382 | value: 0.234 |
| 1383 | - type: precision_at_3 |
| 1384 | value: 24.126 |
| 1385 | - type: precision_at_5 |
| 1386 | value: 17.562 |
| 1387 | - type: recall_at_1 |
| 1388 | value: 19.338 |
| 1389 | - type: recall_at_10 |
| 1390 | value: 45.735 |
| 1391 | - type: recall_at_100 |
| 1392 | value: 71.281 |
| 1393 | - type: recall_at_1000 |
| 1394 | value: 89.537 |
| 1395 | - type: recall_at_3 |
| 1396 | value: 32.525 |
| 1397 | - type: recall_at_5 |
| 1398 | value: 37.671 |
| 1399 | - task: |
| 1400 | type: Retrieval |
| 1401 | dataset: |
| 1402 | type: hotpotqa |
| 1403 | name: MTEB HotpotQA |
| 1404 | config: default |
| 1405 | split: test |
| 1406 | revision: None |
| 1407 | metrics: |
| 1408 | - type: map_at_1 |
| 1409 | value: 36.995 |
| 1410 | - type: map_at_10 |
| 1411 | value: 55.032000000000004 |
| 1412 | - type: map_at_100 |
| 1413 | value: 55.86 |
| 1414 | - type: map_at_1000 |
| 1415 | value: 55.932 |
| 1416 | - type: map_at_3 |
| 1417 | value: 52.125 |
| 1418 | - type: map_at_5 |
| 1419 | value: 53.884 |
| 1420 | - type: mrr_at_1 |
| 1421 | value: 73.991 |
| 1422 | - type: mrr_at_10 |
| 1423 | value: 80.096 |
| 1424 | - type: mrr_at_100 |
| 1425 | value: 80.32000000000001 |
| 1426 | - type: mrr_at_1000 |
| 1427 | value: 80.331 |
| 1428 | - type: mrr_at_3 |
| 1429 | value: 79.037 |
| 1430 | - type: mrr_at_5 |
| 1431 | value: 79.719 |
| 1432 | - type: ndcg_at_1 |
| 1433 | value: 73.991 |
| 1434 | - type: ndcg_at_10 |
| 1435 | value: 63.786 |
| 1436 | - type: ndcg_at_100 |
| 1437 | value: 66.78 |
| 1438 | - type: ndcg_at_1000 |
| 1439 | value: 68.255 |
| 1440 | - type: ndcg_at_3 |
| 1441 | value: 59.501000000000005 |
| 1442 | - type: ndcg_at_5 |
| 1443 | value: 61.82299999999999 |
| 1444 | - type: precision_at_1 |
| 1445 | value: 73.991 |
| 1446 | - type: precision_at_10 |
| 1447 | value: 13.157 |
| 1448 | - type: precision_at_100 |
| 1449 | value: 1.552 |
| 1450 | - type: precision_at_1000 |
| 1451 | value: 0.17500000000000002 |
| 1452 | - type: precision_at_3 |
| 1453 | value: 37.519999999999996 |
| 1454 | - type: precision_at_5 |
| 1455 | value: 24.351 |
| 1456 | - type: recall_at_1 |
| 1457 | value: 36.995 |
| 1458 | - type: recall_at_10 |
| 1459 | value: 65.78699999999999 |
| 1460 | - type: recall_at_100 |
| 1461 | value: 77.583 |
| 1462 | - type: recall_at_1000 |
| 1463 | value: 87.421 |
| 1464 | - type: recall_at_3 |
| 1465 | value: 56.279999999999994 |
| 1466 | - type: recall_at_5 |
| 1467 | value: 60.878 |
| 1468 | - task: |
| 1469 | type: Classification |
| 1470 | dataset: |
| 1471 | type: mteb/imdb |
| 1472 | name: MTEB ImdbClassification |
| 1473 | config: default |
| 1474 | split: test |
| 1475 | revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
| 1476 | metrics: |
| 1477 | - type: accuracy |
| 1478 | value: 86.80239999999999 |
| 1479 | - type: ap |
| 1480 | value: 81.97305141128378 |
| 1481 | - type: f1 |
| 1482 | value: 86.76976305549273 |
| 1483 | - task: |
| 1484 | type: Retrieval |
| 1485 | dataset: |
| 1486 | type: msmarco |
| 1487 | name: MTEB MSMARCO |
| 1488 | config: default |
| 1489 | split: dev |
| 1490 | revision: None |
| 1491 | metrics: |
| 1492 | - type: map_at_1 |
| 1493 | value: 21.166 |
| 1494 | - type: map_at_10 |
| 1495 | value: 33.396 |
| 1496 | - type: map_at_100 |
| 1497 | value: 34.588 |
| 1498 | - type: map_at_1000 |
| 1499 | value: 34.637 |
| 1500 | - type: map_at_3 |
| 1501 | value: 29.509999999999998 |
| 1502 | - type: map_at_5 |
| 1503 | value: 31.719 |
| 1504 | - type: mrr_at_1 |
| 1505 | value: 21.762 |
| 1506 | - type: mrr_at_10 |
| 1507 | value: 33.969 |
| 1508 | - type: mrr_at_100 |
| 1509 | value: 35.099000000000004 |
| 1510 | - type: mrr_at_1000 |
| 1511 | value: 35.141 |
| 1512 | - type: mrr_at_3 |
| 1513 | value: 30.148000000000003 |
| 1514 | - type: mrr_at_5 |
| 1515 | value: 32.324000000000005 |
| 1516 | - type: ndcg_at_1 |
| 1517 | value: 21.776999999999997 |
| 1518 | - type: ndcg_at_10 |
| 1519 | value: 40.306999999999995 |
| 1520 | - type: ndcg_at_100 |
| 1521 | value: 46.068 |
| 1522 | - type: ndcg_at_1000 |
| 1523 | value: 47.3 |
| 1524 | - type: ndcg_at_3 |
| 1525 | value: 32.416 |
| 1526 | - type: ndcg_at_5 |
| 1527 | value: 36.345 |
| 1528 | - type: precision_at_1 |
| 1529 | value: 21.776999999999997 |
| 1530 | - type: precision_at_10 |
| 1531 | value: 6.433 |
| 1532 | - type: precision_at_100 |
| 1533 | value: 0.932 |
| 1534 | - type: precision_at_1000 |
| 1535 | value: 0.104 |
| 1536 | - type: precision_at_3 |
| 1537 | value: 13.897 |
| 1538 | - type: precision_at_5 |
| 1539 | value: 10.324 |
| 1540 | - type: recall_at_1 |
| 1541 | value: 21.166 |
| 1542 | - type: recall_at_10 |
| 1543 | value: 61.587 |
| 1544 | - type: recall_at_100 |
| 1545 | value: 88.251 |
| 1546 | - type: recall_at_1000 |
| 1547 | value: 97.727 |
| 1548 | - type: recall_at_3 |
| 1549 | value: 40.196 |
| 1550 | - type: recall_at_5 |
| 1551 | value: 49.611 |
| 1552 | - task: |
| 1553 | type: Classification |
| 1554 | dataset: |
| 1555 | type: mteb/mtop_domain |
| 1556 | name: MTEB MTOPDomainClassification (en) |
| 1557 | config: en |
| 1558 | split: test |
| 1559 | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| 1560 | metrics: |
| 1561 | - type: accuracy |
| 1562 | value: 93.04605563155496 |
| 1563 | - type: f1 |
| 1564 | value: 92.78007303978372 |
| 1565 | - task: |
| 1566 | type: Classification |
| 1567 | dataset: |
| 1568 | type: mteb/mtop_intent |
| 1569 | name: MTEB MTOPIntentClassification (en) |
| 1570 | config: en |
| 1571 | split: test |
| 1572 | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| 1573 | metrics: |
| 1574 | - type: accuracy |
| 1575 | value: 69.65116279069767 |
| 1576 | - type: f1 |
| 1577 | value: 52.75775172527262 |
| 1578 | - task: |
| 1579 | type: Classification |
| 1580 | dataset: |
| 1581 | type: mteb/amazon_massive_intent |
| 1582 | name: MTEB MassiveIntentClassification (en) |
| 1583 | config: en |
| 1584 | split: test |
| 1585 | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| 1586 | metrics: |
| 1587 | - type: accuracy |
| 1588 | value: 70.34633490248822 |
| 1589 | - type: f1 |
| 1590 | value: 68.15345065392562 |
| 1591 | - task: |
| 1592 | type: Classification |
| 1593 | dataset: |
| 1594 | type: mteb/amazon_massive_scenario |
| 1595 | name: MTEB MassiveScenarioClassification (en) |
| 1596 | config: en |
| 1597 | split: test |
| 1598 | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| 1599 | metrics: |
| 1600 | - type: accuracy |
| 1601 | value: 75.63887020847343 |
| 1602 | - type: f1 |
| 1603 | value: 76.08074680233685 |
| 1604 | - task: |
| 1605 | type: Clustering |
| 1606 | dataset: |
| 1607 | type: mteb/medrxiv-clustering-p2p |
| 1608 | name: MTEB MedrxivClusteringP2P |
| 1609 | config: default |
| 1610 | split: test |
| 1611 | revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
| 1612 | metrics: |
| 1613 | - type: v_measure |
| 1614 | value: 33.77933406071333 |
| 1615 | - task: |
| 1616 | type: Clustering |
| 1617 | dataset: |
| 1618 | type: mteb/medrxiv-clustering-s2s |
| 1619 | name: MTEB MedrxivClusteringS2S |
| 1620 | config: default |
| 1621 | split: test |
| 1622 | revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
| 1623 | metrics: |
| 1624 | - type: v_measure |
| 1625 | value: 32.06504927238196 |
| 1626 | - task: |
| 1627 | type: Reranking |
| 1628 | dataset: |
| 1629 | type: mteb/mind_small |
| 1630 | name: MTEB MindSmallReranking |
| 1631 | config: default |
| 1632 | split: test |
| 1633 | revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
| 1634 | metrics: |
| 1635 | - type: map |
| 1636 | value: 32.20682480490871 |
| 1637 | - type: mrr |
| 1638 | value: 33.41462721527003 |
| 1639 | - task: |
| 1640 | type: Retrieval |
| 1641 | dataset: |
| 1642 | type: nfcorpus |
| 1643 | name: MTEB NFCorpus |
| 1644 | config: default |
| 1645 | split: test |
| 1646 | revision: None |
| 1647 | metrics: |
| 1648 | - type: map_at_1 |
| 1649 | value: 5.548 |
| 1650 | - type: map_at_10 |
| 1651 | value: 13.086999999999998 |
| 1652 | - type: map_at_100 |
| 1653 | value: 16.698 |
| 1654 | - type: map_at_1000 |
| 1655 | value: 18.151999999999997 |
| 1656 | - type: map_at_3 |
| 1657 | value: 9.576 |
| 1658 | - type: map_at_5 |
| 1659 | value: 11.175 |
| 1660 | - type: mrr_at_1 |
| 1661 | value: 44.272 |
| 1662 | - type: mrr_at_10 |
| 1663 | value: 53.635999999999996 |
| 1664 | - type: mrr_at_100 |
| 1665 | value: 54.228 |
| 1666 | - type: mrr_at_1000 |
| 1667 | value: 54.26499999999999 |
| 1668 | - type: mrr_at_3 |
| 1669 | value: 51.754 |
| 1670 | - type: mrr_at_5 |
| 1671 | value: 53.086 |
| 1672 | - type: ndcg_at_1 |
| 1673 | value: 42.724000000000004 |
| 1674 | - type: ndcg_at_10 |
| 1675 | value: 34.769 |
| 1676 | - type: ndcg_at_100 |
| 1677 | value: 32.283 |
| 1678 | - type: ndcg_at_1000 |
| 1679 | value: 40.843 |
| 1680 | - type: ndcg_at_3 |
| 1681 | value: 39.852 |
| 1682 | - type: ndcg_at_5 |
| 1683 | value: 37.858999999999995 |
| 1684 | - type: precision_at_1 |
| 1685 | value: 44.272 |
| 1686 | - type: precision_at_10 |
| 1687 | value: 26.068 |
| 1688 | - type: precision_at_100 |
| 1689 | value: 8.328000000000001 |
| 1690 | - type: precision_at_1000 |
| 1691 | value: 2.1 |
| 1692 | - type: precision_at_3 |
| 1693 | value: 37.874 |
| 1694 | - type: precision_at_5 |
| 1695 | value: 33.065 |
| 1696 | - type: recall_at_1 |
| 1697 | value: 5.548 |
| 1698 | - type: recall_at_10 |
| 1699 | value: 16.936999999999998 |
| 1700 | - type: recall_at_100 |
| 1701 | value: 33.72 |
| 1702 | - type: recall_at_1000 |
| 1703 | value: 64.348 |
| 1704 | - type: recall_at_3 |
| 1705 | value: 10.764999999999999 |
| 1706 | - type: recall_at_5 |
| 1707 | value: 13.361 |
| 1708 | - task: |
| 1709 | type: Retrieval |
| 1710 | dataset: |
| 1711 | type: nq |
| 1712 | name: MTEB NQ |
| 1713 | config: default |
| 1714 | split: test |
| 1715 | revision: None |
| 1716 | metrics: |
| 1717 | - type: map_at_1 |
| 1718 | value: 28.008 |
| 1719 | - type: map_at_10 |
| 1720 | value: 42.675000000000004 |
| 1721 | - type: map_at_100 |
| 1722 | value: 43.85 |
| 1723 | - type: map_at_1000 |
| 1724 | value: 43.884 |
| 1725 | - type: map_at_3 |
| 1726 | value: 38.286 |
| 1727 | - type: map_at_5 |
| 1728 | value: 40.78 |
| 1729 | - type: mrr_at_1 |
| 1730 | value: 31.518 |
| 1731 | - type: mrr_at_10 |
| 1732 | value: 45.015 |
| 1733 | - type: mrr_at_100 |
| 1734 | value: 45.924 |
| 1735 | - type: mrr_at_1000 |
| 1736 | value: 45.946999999999996 |
| 1737 | - type: mrr_at_3 |
| 1738 | value: 41.348 |
| 1739 | - type: mrr_at_5 |
| 1740 | value: 43.428 |
| 1741 | - type: ndcg_at_1 |
| 1742 | value: 31.489 |
| 1743 | - type: ndcg_at_10 |
| 1744 | value: 50.285999999999994 |
| 1745 | - type: ndcg_at_100 |
| 1746 | value: 55.291999999999994 |
| 1747 | - type: ndcg_at_1000 |
| 1748 | value: 56.05 |
| 1749 | - type: ndcg_at_3 |
| 1750 | value: 41.976 |
| 1751 | - type: ndcg_at_5 |
| 1752 | value: 46.103 |
| 1753 | - type: precision_at_1 |
| 1754 | value: 31.489 |
| 1755 | - type: precision_at_10 |
| 1756 | value: 8.456 |
| 1757 | - type: precision_at_100 |
| 1758 | value: 1.125 |
| 1759 | - type: precision_at_1000 |
| 1760 | value: 0.12 |
| 1761 | - type: precision_at_3 |
| 1762 | value: 19.09 |
| 1763 | - type: precision_at_5 |
| 1764 | value: 13.841000000000001 |
| 1765 | - type: recall_at_1 |
| 1766 | value: 28.008 |
| 1767 | - type: recall_at_10 |
| 1768 | value: 71.21499999999999 |
| 1769 | - type: recall_at_100 |
| 1770 | value: 92.99 |
| 1771 | - type: recall_at_1000 |
| 1772 | value: 98.578 |
| 1773 | - type: recall_at_3 |
| 1774 | value: 49.604 |
| 1775 | - type: recall_at_5 |
| 1776 | value: 59.094 |
| 1777 | - task: |
| 1778 | type: Retrieval |
| 1779 | dataset: |
| 1780 | type: quora |
| 1781 | name: MTEB QuoraRetrieval |
| 1782 | config: default |
| 1783 | split: test |
| 1784 | revision: None |
| 1785 | metrics: |
| 1786 | - type: map_at_1 |
| 1787 | value: 70.351 |
| 1788 | - type: map_at_10 |
| 1789 | value: 84.163 |
| 1790 | - type: map_at_100 |
| 1791 | value: 84.785 |
| 1792 | - type: map_at_1000 |
| 1793 | value: 84.801 |
| 1794 | - type: map_at_3 |
| 1795 | value: 81.16 |
| 1796 | - type: map_at_5 |
| 1797 | value: 83.031 |
| 1798 | - type: mrr_at_1 |
| 1799 | value: 80.96 |
| 1800 | - type: mrr_at_10 |
| 1801 | value: 87.241 |
| 1802 | - type: mrr_at_100 |
| 1803 | value: 87.346 |
| 1804 | - type: mrr_at_1000 |
| 1805 | value: 87.347 |
| 1806 | - type: mrr_at_3 |
| 1807 | value: 86.25699999999999 |
| 1808 | - type: mrr_at_5 |
| 1809 | value: 86.907 |
| 1810 | - type: ndcg_at_1 |
| 1811 | value: 80.97 |
| 1812 | - type: ndcg_at_10 |
| 1813 | value: 88.017 |
| 1814 | - type: ndcg_at_100 |
| 1815 | value: 89.241 |
| 1816 | - type: ndcg_at_1000 |
| 1817 | value: 89.34299999999999 |
| 1818 | - type: ndcg_at_3 |
| 1819 | value: 85.053 |
| 1820 | - type: ndcg_at_5 |
| 1821 | value: 86.663 |
| 1822 | - type: precision_at_1 |
| 1823 | value: 80.97 |
| 1824 | - type: precision_at_10 |
| 1825 | value: 13.358 |
| 1826 | - type: precision_at_100 |
| 1827 | value: 1.525 |
| 1828 | - type: precision_at_1000 |
| 1829 | value: 0.157 |
| 1830 | - type: precision_at_3 |
| 1831 | value: 37.143 |
| 1832 | - type: precision_at_5 |
| 1833 | value: 24.451999999999998 |
| 1834 | - type: recall_at_1 |
| 1835 | value: 70.351 |
| 1836 | - type: recall_at_10 |
| 1837 | value: 95.39800000000001 |
| 1838 | - type: recall_at_100 |
| 1839 | value: 99.55199999999999 |
| 1840 | - type: recall_at_1000 |
| 1841 | value: 99.978 |
| 1842 | - type: recall_at_3 |
| 1843 | value: 86.913 |
| 1844 | - type: recall_at_5 |
| 1845 | value: 91.448 |
| 1846 | - task: |
| 1847 | type: Clustering |
| 1848 | dataset: |
| 1849 | type: mteb/reddit-clustering |
| 1850 | name: MTEB RedditClustering |
| 1851 | config: default |
| 1852 | split: test |
| 1853 | revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
| 1854 | metrics: |
| 1855 | - type: v_measure |
| 1856 | value: 55.62406719814139 |
| 1857 | - task: |
| 1858 | type: Clustering |
| 1859 | dataset: |
| 1860 | type: mteb/reddit-clustering-p2p |
| 1861 | name: MTEB RedditClusteringP2P |
| 1862 | config: default |
| 1863 | split: test |
| 1864 | revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
| 1865 | metrics: |
| 1866 | - type: v_measure |
| 1867 | value: 61.386700035141736 |
| 1868 | - task: |
| 1869 | type: Retrieval |
| 1870 | dataset: |
| 1871 | type: scidocs |
| 1872 | name: MTEB SCIDOCS |
| 1873 | config: default |
| 1874 | split: test |
| 1875 | revision: None |
| 1876 | metrics: |
| 1877 | - type: map_at_1 |
| 1878 | value: 4.618 |
| 1879 | - type: map_at_10 |
| 1880 | value: 12.920000000000002 |
| 1881 | - type: map_at_100 |
| 1882 | value: 15.304 |
| 1883 | - type: map_at_1000 |
| 1884 | value: 15.656999999999998 |
| 1885 | - type: map_at_3 |
| 1886 | value: 9.187 |
| 1887 | - type: map_at_5 |
| 1888 | value: 10.937 |
| 1889 | - type: mrr_at_1 |
| 1890 | value: 22.8 |
| 1891 | - type: mrr_at_10 |
| 1892 | value: 35.13 |
| 1893 | - type: mrr_at_100 |
| 1894 | value: 36.239 |
| 1895 | - type: mrr_at_1000 |
| 1896 | value: 36.291000000000004 |
| 1897 | - type: mrr_at_3 |
| 1898 | value: 31.917 |
| 1899 | - type: mrr_at_5 |
| 1900 | value: 33.787 |
| 1901 | - type: ndcg_at_1 |
| 1902 | value: 22.8 |
| 1903 | - type: ndcg_at_10 |
| 1904 | value: 21.382 |
| 1905 | - type: ndcg_at_100 |
| 1906 | value: 30.257 |
| 1907 | - type: ndcg_at_1000 |
| 1908 | value: 36.001 |
| 1909 | - type: ndcg_at_3 |
| 1910 | value: 20.43 |
| 1911 | - type: ndcg_at_5 |
| 1912 | value: 17.622 |
| 1913 | - type: precision_at_1 |
| 1914 | value: 22.8 |
| 1915 | - type: precision_at_10 |
| 1916 | value: 11.26 |
| 1917 | - type: precision_at_100 |
| 1918 | value: 2.405 |
| 1919 | - type: precision_at_1000 |
| 1920 | value: 0.377 |
| 1921 | - type: precision_at_3 |
| 1922 | value: 19.633 |
| 1923 | - type: precision_at_5 |
| 1924 | value: 15.68 |
| 1925 | - type: recall_at_1 |
| 1926 | value: 4.618 |
| 1927 | - type: recall_at_10 |
| 1928 | value: 22.811999999999998 |
| 1929 | - type: recall_at_100 |
| 1930 | value: 48.787000000000006 |
| 1931 | - type: recall_at_1000 |
| 1932 | value: 76.63799999999999 |
| 1933 | - type: recall_at_3 |
| 1934 | value: 11.952 |
| 1935 | - type: recall_at_5 |
| 1936 | value: 15.892000000000001 |
| 1937 | - task: |
| 1938 | type: STS |
| 1939 | dataset: |
| 1940 | type: mteb/sickr-sts |
| 1941 | name: MTEB SICK-R |
| 1942 | config: default |
| 1943 | split: test |
| 1944 | revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
| 1945 | metrics: |
| 1946 | - type: cos_sim_pearson |
| 1947 | value: 84.01529458252244 |
| 1948 | - type: cos_sim_spearman |
| 1949 | value: 77.92985224770254 |
| 1950 | - type: euclidean_pearson |
| 1951 | value: 81.04251429422487 |
| 1952 | - type: euclidean_spearman |
| 1953 | value: 77.92838490549133 |
| 1954 | - type: manhattan_pearson |
| 1955 | value: 80.95892251458979 |
| 1956 | - type: manhattan_spearman |
| 1957 | value: 77.81028089705941 |
| 1958 | - task: |
| 1959 | type: STS |
| 1960 | dataset: |
| 1961 | type: mteb/sts12-sts |
| 1962 | name: MTEB STS12 |
| 1963 | config: default |
| 1964 | split: test |
| 1965 | revision: a0d554a64d88156834ff5ae9920b964011b16384 |
| 1966 | metrics: |
| 1967 | - type: cos_sim_pearson |
| 1968 | value: 83.97885282534388 |
| 1969 | - type: cos_sim_spearman |
| 1970 | value: 75.1221970851712 |
| 1971 | - type: euclidean_pearson |
| 1972 | value: 80.34455956720097 |
| 1973 | - type: euclidean_spearman |
| 1974 | value: 74.5894274239938 |
| 1975 | - type: manhattan_pearson |
| 1976 | value: 80.38999766325465 |
| 1977 | - type: manhattan_spearman |
| 1978 | value: 74.68524557166975 |
| 1979 | - task: |
| 1980 | type: STS |
| 1981 | dataset: |
| 1982 | type: mteb/sts13-sts |
| 1983 | name: MTEB STS13 |
| 1984 | config: default |
| 1985 | split: test |
| 1986 | revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
| 1987 | metrics: |
| 1988 | - type: cos_sim_pearson |
| 1989 | value: 82.95746064915672 |
| 1990 | - type: cos_sim_spearman |
| 1991 | value: 85.08683458043946 |
| 1992 | - type: euclidean_pearson |
| 1993 | value: 84.56699492836385 |
| 1994 | - type: euclidean_spearman |
| 1995 | value: 85.66089116133713 |
| 1996 | - type: manhattan_pearson |
| 1997 | value: 84.47553323458541 |
| 1998 | - type: manhattan_spearman |
| 1999 | value: 85.56142206781472 |
| 2000 | - task: |
| 2001 | type: STS |
| 2002 | dataset: |
| 2003 | type: mteb/sts14-sts |
| 2004 | name: MTEB STS14 |
| 2005 | config: default |
| 2006 | split: test |
| 2007 | revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
| 2008 | metrics: |
| 2009 | - type: cos_sim_pearson |
| 2010 | value: 82.71377893595067 |
| 2011 | - type: cos_sim_spearman |
| 2012 | value: 81.03453291428589 |
| 2013 | - type: euclidean_pearson |
| 2014 | value: 82.57136298308613 |
| 2015 | - type: euclidean_spearman |
| 2016 | value: 81.15839961890875 |
| 2017 | - type: manhattan_pearson |
| 2018 | value: 82.55157879373837 |
| 2019 | - type: manhattan_spearman |
| 2020 | value: 81.1540163767054 |
| 2021 | - task: |
| 2022 | type: STS |
| 2023 | dataset: |
| 2024 | type: mteb/sts15-sts |
| 2025 | name: MTEB STS15 |
| 2026 | config: default |
| 2027 | split: test |
| 2028 | revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
| 2029 | metrics: |
| 2030 | - type: cos_sim_pearson |
| 2031 | value: 86.64197832372373 |
| 2032 | - type: cos_sim_spearman |
| 2033 | value: 88.31966852492485 |
| 2034 | - type: euclidean_pearson |
| 2035 | value: 87.98692129976983 |
| 2036 | - type: euclidean_spearman |
| 2037 | value: 88.6247340837856 |
| 2038 | - type: manhattan_pearson |
| 2039 | value: 87.90437827826412 |
| 2040 | - type: manhattan_spearman |
| 2041 | value: 88.56278787131457 |
| 2042 | - task: |
| 2043 | type: STS |
| 2044 | dataset: |
| 2045 | type: mteb/sts16-sts |
| 2046 | name: MTEB STS16 |
| 2047 | config: default |
| 2048 | split: test |
| 2049 | revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
| 2050 | metrics: |
| 2051 | - type: cos_sim_pearson |
| 2052 | value: 81.84159950146693 |
| 2053 | - type: cos_sim_spearman |
| 2054 | value: 83.90678384140168 |
| 2055 | - type: euclidean_pearson |
| 2056 | value: 83.19005018860221 |
| 2057 | - type: euclidean_spearman |
| 2058 | value: 84.16260415876295 |
| 2059 | - type: manhattan_pearson |
| 2060 | value: 83.05030612994494 |
| 2061 | - type: manhattan_spearman |
| 2062 | value: 83.99605629718336 |
| 2063 | - task: |
| 2064 | type: STS |
| 2065 | dataset: |
| 2066 | type: mteb/sts17-crosslingual-sts |
| 2067 | name: MTEB STS17 (en-en) |
| 2068 | config: en-en |
| 2069 | split: test |
| 2070 | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| 2071 | metrics: |
| 2072 | - type: cos_sim_pearson |
| 2073 | value: 87.49935350176666 |
| 2074 | - type: cos_sim_spearman |
| 2075 | value: 87.59086606735383 |
| 2076 | - type: euclidean_pearson |
| 2077 | value: 88.06537181129983 |
| 2078 | - type: euclidean_spearman |
| 2079 | value: 87.6687448086014 |
| 2080 | - type: manhattan_pearson |
| 2081 | value: 87.96599131972935 |
| 2082 | - type: manhattan_spearman |
| 2083 | value: 87.63295748969642 |
| 2084 | - task: |
| 2085 | type: STS |
| 2086 | dataset: |
| 2087 | type: mteb/sts22-crosslingual-sts |
| 2088 | name: MTEB STS22 (en) |
| 2089 | config: en |
| 2090 | split: test |
| 2091 | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| 2092 | metrics: |
| 2093 | - type: cos_sim_pearson |
| 2094 | value: 67.68232799482763 |
| 2095 | - type: cos_sim_spearman |
| 2096 | value: 67.99930378085793 |
| 2097 | - type: euclidean_pearson |
| 2098 | value: 68.50275360001696 |
| 2099 | - type: euclidean_spearman |
| 2100 | value: 67.81588179309259 |
| 2101 | - type: manhattan_pearson |
| 2102 | value: 68.5892154749763 |
| 2103 | - type: manhattan_spearman |
| 2104 | value: 67.84357259640682 |
| 2105 | - task: |
| 2106 | type: STS |
| 2107 | dataset: |
| 2108 | type: mteb/stsbenchmark-sts |
| 2109 | name: MTEB STSBenchmark |
| 2110 | config: default |
| 2111 | split: test |
| 2112 | revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
| 2113 | metrics: |
| 2114 | - type: cos_sim_pearson |
| 2115 | value: 84.37049618406554 |
| 2116 | - type: cos_sim_spearman |
| 2117 | value: 85.57014313159492 |
| 2118 | - type: euclidean_pearson |
| 2119 | value: 85.57469513908282 |
| 2120 | - type: euclidean_spearman |
| 2121 | value: 85.661948135258 |
| 2122 | - type: manhattan_pearson |
| 2123 | value: 85.36866831229028 |
| 2124 | - type: manhattan_spearman |
| 2125 | value: 85.5043455368843 |
| 2126 | - task: |
| 2127 | type: Reranking |
| 2128 | dataset: |
| 2129 | type: mteb/scidocs-reranking |
| 2130 | name: MTEB SciDocsRR |
| 2131 | config: default |
| 2132 | split: test |
| 2133 | revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
| 2134 | metrics: |
| 2135 | - type: map |
| 2136 | value: 84.83259065376154 |
| 2137 | - type: mrr |
| 2138 | value: 95.58455433455433 |
| 2139 | - task: |
| 2140 | type: Retrieval |
| 2141 | dataset: |
| 2142 | type: scifact |
| 2143 | name: MTEB SciFact |
| 2144 | config: default |
| 2145 | split: test |
| 2146 | revision: None |
| 2147 | metrics: |
| 2148 | - type: map_at_1 |
| 2149 | value: 58.817 |
| 2150 | - type: map_at_10 |
| 2151 | value: 68.459 |
| 2152 | - type: map_at_100 |
| 2153 | value: 68.951 |
| 2154 | - type: map_at_1000 |
| 2155 | value: 68.979 |
| 2156 | - type: map_at_3 |
| 2157 | value: 65.791 |
| 2158 | - type: map_at_5 |
| 2159 | value: 67.583 |
| 2160 | - type: mrr_at_1 |
| 2161 | value: 61.667 |
| 2162 | - type: mrr_at_10 |
| 2163 | value: 69.368 |
| 2164 | - type: mrr_at_100 |
| 2165 | value: 69.721 |
| 2166 | - type: mrr_at_1000 |
| 2167 | value: 69.744 |
| 2168 | - type: mrr_at_3 |
| 2169 | value: 67.278 |
| 2170 | - type: mrr_at_5 |
| 2171 | value: 68.611 |
| 2172 | - type: ndcg_at_1 |
| 2173 | value: 61.667 |
| 2174 | - type: ndcg_at_10 |
| 2175 | value: 72.70100000000001 |
| 2176 | - type: ndcg_at_100 |
| 2177 | value: 74.928 |
| 2178 | - type: ndcg_at_1000 |
| 2179 | value: 75.553 |
| 2180 | - type: ndcg_at_3 |
| 2181 | value: 68.203 |
| 2182 | - type: ndcg_at_5 |
| 2183 | value: 70.804 |
| 2184 | - type: precision_at_1 |
| 2185 | value: 61.667 |
| 2186 | - type: precision_at_10 |
| 2187 | value: 9.533 |
| 2188 | - type: precision_at_100 |
| 2189 | value: 1.077 |
| 2190 | - type: precision_at_1000 |
| 2191 | value: 0.11299999999999999 |
| 2192 | - type: precision_at_3 |
| 2193 | value: 26.444000000000003 |
| 2194 | - type: precision_at_5 |
| 2195 | value: 17.599999999999998 |
| 2196 | - type: recall_at_1 |
| 2197 | value: 58.817 |
| 2198 | - type: recall_at_10 |
| 2199 | value: 84.789 |
| 2200 | - type: recall_at_100 |
| 2201 | value: 95.0 |
| 2202 | - type: recall_at_1000 |
| 2203 | value: 99.667 |
| 2204 | - type: recall_at_3 |
| 2205 | value: 72.8 |
| 2206 | - type: recall_at_5 |
| 2207 | value: 79.294 |
| 2208 | - task: |
| 2209 | type: PairClassification |
| 2210 | dataset: |
| 2211 | type: mteb/sprintduplicatequestions-pairclassification |
| 2212 | name: MTEB SprintDuplicateQuestions |
| 2213 | config: default |
| 2214 | split: test |
| 2215 | revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
| 2216 | metrics: |
| 2217 | - type: cos_sim_accuracy |
| 2218 | value: 99.8108910891089 |
| 2219 | - type: cos_sim_ap |
| 2220 | value: 95.5743678558349 |
| 2221 | - type: cos_sim_f1 |
| 2222 | value: 90.43133366385722 |
| 2223 | - type: cos_sim_precision |
| 2224 | value: 89.67551622418878 |
| 2225 | - type: cos_sim_recall |
| 2226 | value: 91.2 |
| 2227 | - type: dot_accuracy |
| 2228 | value: 99.75841584158415 |
| 2229 | - type: dot_ap |
| 2230 | value: 94.00786363627253 |
| 2231 | - type: dot_f1 |
| 2232 | value: 87.51910341314316 |
| 2233 | - type: dot_precision |
| 2234 | value: 89.20041536863967 |
| 2235 | - type: dot_recall |
| 2236 | value: 85.9 |
| 2237 | - type: euclidean_accuracy |
| 2238 | value: 99.81485148514851 |
| 2239 | - type: euclidean_ap |
| 2240 | value: 95.4752113136905 |
| 2241 | - type: euclidean_f1 |
| 2242 | value: 90.44334975369456 |
| 2243 | - type: euclidean_precision |
| 2244 | value: 89.126213592233 |
| 2245 | - type: euclidean_recall |
| 2246 | value: 91.8 |
| 2247 | - type: manhattan_accuracy |
| 2248 | value: 99.81584158415842 |
| 2249 | - type: manhattan_ap |
| 2250 | value: 95.5163172682464 |
| 2251 | - type: manhattan_f1 |
| 2252 | value: 90.51987767584097 |
| 2253 | - type: manhattan_precision |
| 2254 | value: 92.3076923076923 |
| 2255 | - type: manhattan_recall |
| 2256 | value: 88.8 |
| 2257 | - type: max_accuracy |
| 2258 | value: 99.81584158415842 |
| 2259 | - type: max_ap |
| 2260 | value: 95.5743678558349 |
| 2261 | - type: max_f1 |
| 2262 | value: 90.51987767584097 |
| 2263 | - task: |
| 2264 | type: Clustering |
| 2265 | dataset: |
| 2266 | type: mteb/stackexchange-clustering |
| 2267 | name: MTEB StackExchangeClustering |
| 2268 | config: default |
| 2269 | split: test |
| 2270 | revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
| 2271 | metrics: |
| 2272 | - type: v_measure |
| 2273 | value: 62.63235986949449 |
| 2274 | - task: |
| 2275 | type: Clustering |
| 2276 | dataset: |
| 2277 | type: mteb/stackexchange-clustering-p2p |
| 2278 | name: MTEB StackExchangeClusteringP2P |
| 2279 | config: default |
| 2280 | split: test |
| 2281 | revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
| 2282 | metrics: |
| 2283 | - type: v_measure |
| 2284 | value: 36.334795589585575 |
| 2285 | - task: |
| 2286 | type: Reranking |
| 2287 | dataset: |
| 2288 | type: mteb/stackoverflowdupquestions-reranking |
| 2289 | name: MTEB StackOverflowDupQuestions |
| 2290 | config: default |
| 2291 | split: test |
| 2292 | revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
| 2293 | metrics: |
| 2294 | - type: map |
| 2295 | value: 52.02955214518782 |
| 2296 | - type: mrr |
| 2297 | value: 52.8004838298956 |
| 2298 | - task: |
| 2299 | type: Summarization |
| 2300 | dataset: |
| 2301 | type: mteb/summeval |
| 2302 | name: MTEB SummEval |
| 2303 | config: default |
| 2304 | split: test |
| 2305 | revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
| 2306 | metrics: |
| 2307 | - type: cos_sim_pearson |
| 2308 | value: 30.63769566275453 |
| 2309 | - type: cos_sim_spearman |
| 2310 | value: 30.422379185989335 |
| 2311 | - type: dot_pearson |
| 2312 | value: 26.88493071882256 |
| 2313 | - type: dot_spearman |
| 2314 | value: 26.505249740971305 |
| 2315 | - task: |
| 2316 | type: Retrieval |
| 2317 | dataset: |
| 2318 | type: trec-covid |
| 2319 | name: MTEB TRECCOVID |
| 2320 | config: default |
| 2321 | split: test |
| 2322 | revision: None |
| 2323 | metrics: |
| 2324 | - type: map_at_1 |
| 2325 | value: 0.21 |
| 2326 | - type: map_at_10 |
| 2327 | value: 1.654 |
| 2328 | - type: map_at_100 |
| 2329 | value: 10.095 |
| 2330 | - type: map_at_1000 |
| 2331 | value: 25.808999999999997 |
| 2332 | - type: map_at_3 |
| 2333 | value: 0.594 |
| 2334 | - type: map_at_5 |
| 2335 | value: 0.9289999999999999 |
| 2336 | - type: mrr_at_1 |
| 2337 | value: 78.0 |
| 2338 | - type: mrr_at_10 |
| 2339 | value: 87.019 |
| 2340 | - type: mrr_at_100 |
| 2341 | value: 87.019 |
| 2342 | - type: mrr_at_1000 |
| 2343 | value: 87.019 |
| 2344 | - type: mrr_at_3 |
| 2345 | value: 86.333 |
| 2346 | - type: mrr_at_5 |
| 2347 | value: 86.733 |
| 2348 | - type: ndcg_at_1 |
| 2349 | value: 73.0 |
| 2350 | - type: ndcg_at_10 |
| 2351 | value: 66.52900000000001 |
| 2352 | - type: ndcg_at_100 |
| 2353 | value: 53.433 |
| 2354 | - type: ndcg_at_1000 |
| 2355 | value: 51.324000000000005 |
| 2356 | - type: ndcg_at_3 |
| 2357 | value: 72.02199999999999 |
| 2358 | - type: ndcg_at_5 |
| 2359 | value: 69.696 |
| 2360 | - type: precision_at_1 |
| 2361 | value: 78.0 |
| 2362 | - type: precision_at_10 |
| 2363 | value: 70.39999999999999 |
| 2364 | - type: precision_at_100 |
| 2365 | value: 55.46 |
| 2366 | - type: precision_at_1000 |
| 2367 | value: 22.758 |
| 2368 | - type: precision_at_3 |
| 2369 | value: 76.667 |
| 2370 | - type: precision_at_5 |
| 2371 | value: 74.0 |
| 2372 | - type: recall_at_1 |
| 2373 | value: 0.21 |
| 2374 | - type: recall_at_10 |
| 2375 | value: 1.8849999999999998 |
| 2376 | - type: recall_at_100 |
| 2377 | value: 13.801 |
| 2378 | - type: recall_at_1000 |
| 2379 | value: 49.649 |
| 2380 | - type: recall_at_3 |
| 2381 | value: 0.632 |
| 2382 | - type: recall_at_5 |
| 2383 | value: 1.009 |
| 2384 | - task: |
| 2385 | type: Retrieval |
| 2386 | dataset: |
| 2387 | type: webis-touche2020 |
| 2388 | name: MTEB Touche2020 |
| 2389 | config: default |
| 2390 | split: test |
| 2391 | revision: None |
| 2392 | metrics: |
| 2393 | - type: map_at_1 |
| 2394 | value: 1.797 |
| 2395 | - type: map_at_10 |
| 2396 | value: 9.01 |
| 2397 | - type: map_at_100 |
| 2398 | value: 14.682 |
| 2399 | - type: map_at_1000 |
| 2400 | value: 16.336000000000002 |
| 2401 | - type: map_at_3 |
| 2402 | value: 4.546 |
| 2403 | - type: map_at_5 |
| 2404 | value: 5.9270000000000005 |
| 2405 | - type: mrr_at_1 |
| 2406 | value: 24.490000000000002 |
| 2407 | - type: mrr_at_10 |
| 2408 | value: 41.156 |
| 2409 | - type: mrr_at_100 |
| 2410 | value: 42.392 |
| 2411 | - type: mrr_at_1000 |
| 2412 | value: 42.408 |
| 2413 | - type: mrr_at_3 |
| 2414 | value: 38.775999999999996 |
| 2415 | - type: mrr_at_5 |
| 2416 | value: 40.102 |
| 2417 | - type: ndcg_at_1 |
| 2418 | value: 21.429000000000002 |
| 2419 | - type: ndcg_at_10 |
| 2420 | value: 22.222 |
| 2421 | - type: ndcg_at_100 |
| 2422 | value: 34.405 |
| 2423 | - type: ndcg_at_1000 |
| 2424 | value: 46.599000000000004 |
| 2425 | - type: ndcg_at_3 |
| 2426 | value: 25.261 |
| 2427 | - type: ndcg_at_5 |
| 2428 | value: 22.695999999999998 |
| 2429 | - type: precision_at_1 |
| 2430 | value: 24.490000000000002 |
| 2431 | - type: precision_at_10 |
| 2432 | value: 19.796 |
| 2433 | - type: precision_at_100 |
| 2434 | value: 7.306 |
| 2435 | - type: precision_at_1000 |
| 2436 | value: 1.5350000000000001 |
| 2437 | - type: precision_at_3 |
| 2438 | value: 27.211000000000002 |
| 2439 | - type: precision_at_5 |
| 2440 | value: 22.857 |
| 2441 | - type: recall_at_1 |
| 2442 | value: 1.797 |
| 2443 | - type: recall_at_10 |
| 2444 | value: 15.706000000000001 |
| 2445 | - type: recall_at_100 |
| 2446 | value: 46.412 |
| 2447 | - type: recall_at_1000 |
| 2448 | value: 83.159 |
| 2449 | - type: recall_at_3 |
| 2450 | value: 6.1370000000000005 |
| 2451 | - type: recall_at_5 |
| 2452 | value: 8.599 |
| 2453 | - task: |
| 2454 | type: Classification |
| 2455 | dataset: |
| 2456 | type: mteb/toxic_conversations_50k |
| 2457 | name: MTEB ToxicConversationsClassification |
| 2458 | config: default |
| 2459 | split: test |
| 2460 | revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
| 2461 | metrics: |
| 2462 | - type: accuracy |
| 2463 | value: 70.3302 |
| 2464 | - type: ap |
| 2465 | value: 14.169121204575601 |
| 2466 | - type: f1 |
| 2467 | value: 54.229345975274235 |
| 2468 | - task: |
| 2469 | type: Classification |
| 2470 | dataset: |
| 2471 | type: mteb/tweet_sentiment_extraction |
| 2472 | name: MTEB TweetSentimentExtractionClassification |
| 2473 | config: default |
| 2474 | split: test |
| 2475 | revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
| 2476 | metrics: |
| 2477 | - type: accuracy |
| 2478 | value: 58.22297679683077 |
| 2479 | - type: f1 |
| 2480 | value: 58.62984908377875 |
| 2481 | - task: |
| 2482 | type: Clustering |
| 2483 | dataset: |
| 2484 | type: mteb/twentynewsgroups-clustering |
| 2485 | name: MTEB TwentyNewsgroupsClustering |
| 2486 | config: default |
| 2487 | split: test |
| 2488 | revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
| 2489 | metrics: |
| 2490 | - type: v_measure |
| 2491 | value: 49.952922428464255 |
| 2492 | - task: |
| 2493 | type: PairClassification |
| 2494 | dataset: |
| 2495 | type: mteb/twittersemeval2015-pairclassification |
| 2496 | name: MTEB TwitterSemEval2015 |
| 2497 | config: default |
| 2498 | split: test |
| 2499 | revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
| 2500 | metrics: |
| 2501 | - type: cos_sim_accuracy |
| 2502 | value: 84.68140907194373 |
| 2503 | - type: cos_sim_ap |
| 2504 | value: 70.12180123666836 |
| 2505 | - type: cos_sim_f1 |
| 2506 | value: 65.77501791258658 |
| 2507 | - type: cos_sim_precision |
| 2508 | value: 60.07853403141361 |
| 2509 | - type: cos_sim_recall |
| 2510 | value: 72.66490765171504 |
| 2511 | - type: dot_accuracy |
| 2512 | value: 81.92167848840674 |
| 2513 | - type: dot_ap |
| 2514 | value: 60.49837581423469 |
| 2515 | - type: dot_f1 |
| 2516 | value: 58.44186046511628 |
| 2517 | - type: dot_precision |
| 2518 | value: 52.24532224532224 |
| 2519 | - type: dot_recall |
| 2520 | value: 66.3060686015831 |
| 2521 | - type: euclidean_accuracy |
| 2522 | value: 84.73505394289802 |
| 2523 | - type: euclidean_ap |
| 2524 | value: 70.3278904593286 |
| 2525 | - type: euclidean_f1 |
| 2526 | value: 65.98851124940161 |
| 2527 | - type: euclidean_precision |
| 2528 | value: 60.38107752956636 |
| 2529 | - type: euclidean_recall |
| 2530 | value: 72.74406332453826 |
| 2531 | - type: manhattan_accuracy |
| 2532 | value: 84.73505394289802 |
| 2533 | - type: manhattan_ap |
| 2534 | value: 70.00737738537337 |
| 2535 | - type: manhattan_f1 |
| 2536 | value: 65.80150784822642 |
| 2537 | - type: manhattan_precision |
| 2538 | value: 61.892583120204606 |
| 2539 | - type: manhattan_recall |
| 2540 | value: 70.23746701846966 |
| 2541 | - type: max_accuracy |
| 2542 | value: 84.73505394289802 |
| 2543 | - type: max_ap |
| 2544 | value: 70.3278904593286 |
| 2545 | - type: max_f1 |
| 2546 | value: 65.98851124940161 |
| 2547 | - task: |
| 2548 | type: PairClassification |
| 2549 | dataset: |
| 2550 | type: mteb/twitterurlcorpus-pairclassification |
| 2551 | name: MTEB TwitterURLCorpus |
| 2552 | config: default |
| 2553 | split: test |
| 2554 | revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
| 2555 | metrics: |
| 2556 | - type: cos_sim_accuracy |
| 2557 | value: 88.44258159661582 |
| 2558 | - type: cos_sim_ap |
| 2559 | value: 84.91926704880888 |
| 2560 | - type: cos_sim_f1 |
| 2561 | value: 77.07651086632926 |
| 2562 | - type: cos_sim_precision |
| 2563 | value: 74.5894554883319 |
| 2564 | - type: cos_sim_recall |
| 2565 | value: 79.73514012935017 |
| 2566 | - type: dot_accuracy |
| 2567 | value: 85.88116583226608 |
| 2568 | - type: dot_ap |
| 2569 | value: 78.9753854779923 |
| 2570 | - type: dot_f1 |
| 2571 | value: 72.17757637979255 |
| 2572 | - type: dot_precision |
| 2573 | value: 66.80647486729143 |
| 2574 | - type: dot_recall |
| 2575 | value: 78.48783492454572 |
| 2576 | - type: euclidean_accuracy |
| 2577 | value: 88.5299025885823 |
| 2578 | - type: euclidean_ap |
| 2579 | value: 85.08006075642194 |
| 2580 | - type: euclidean_f1 |
| 2581 | value: 77.29637336504163 |
| 2582 | - type: euclidean_precision |
| 2583 | value: 74.69836253950014 |
| 2584 | - type: euclidean_recall |
| 2585 | value: 80.08161379735141 |
| 2586 | - type: manhattan_accuracy |
| 2587 | value: 88.55124771995187 |
| 2588 | - type: manhattan_ap |
| 2589 | value: 85.00941529932851 |
| 2590 | - type: manhattan_f1 |
| 2591 | value: 77.33100233100232 |
| 2592 | - type: manhattan_precision |
| 2593 | value: 73.37572573956317 |
| 2594 | - type: manhattan_recall |
| 2595 | value: 81.73698798891284 |
| 2596 | - type: max_accuracy |
| 2597 | value: 88.55124771995187 |
| 2598 | - type: max_ap |
| 2599 | value: 85.08006075642194 |
| 2600 | - type: max_f1 |
| 2601 | value: 77.33100233100232 |
| 2602 | language: |
| 2603 | - en |
| 2604 | license: mit |
| 2605 | --- |
| 2606 | |
| 2607 | # gte-small |
| 2608 | |
| 2609 | General Text Embeddings (GTE) model. [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://arxiv.org/abs/2308.03281) |
| 2610 | |
| 2611 | The GTE models are trained by Alibaba DAMO Academy. They are mainly based on the BERT framework and currently offer three different sizes of models, including [GTE-large](https://huggingface.co/thenlper/gte-large), [GTE-base](https://huggingface.co/thenlper/gte-base), and [GTE-small](https://huggingface.co/thenlper/gte-small). The GTE models are trained on a large-scale corpus of relevance text pairs, covering a wide range of domains and scenarios. This enables the GTE models to be applied to various downstream tasks of text embeddings, including **information retrieval**, **semantic textual similarity**, **text reranking**, etc. |
| 2612 | |
| 2613 | ## Metrics |
| 2614 | |
| 2615 | We compared the performance of the GTE models with other popular text embedding models on the MTEB benchmark. For more detailed comparison results, please refer to the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard). |
| 2616 | |
| 2617 | |
| 2618 | |
| 2619 | | Model Name | Model Size (GB) | Dimension | Sequence Length | Average (56) | Clustering (11) | Pair Classification (3) | Reranking (4) | Retrieval (15) | STS (10) | Summarization (1) | Classification (12) | |
| 2620 | |:----:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |
| 2621 | | [**gte-large**](https://huggingface.co/thenlper/gte-large) | 0.67 | 1024 | 512 | **63.13** | 46.84 | 85.00 | 59.13 | 52.22 | 83.35 | 31.66 | 73.33 | |
| 2622 | | [**gte-base**](https://huggingface.co/thenlper/gte-base) | 0.22 | 768 | 512 | **62.39** | 46.2 | 84.57 | 58.61 | 51.14 | 82.3 | 31.17 | 73.01 | |
| 2623 | | [e5-large-v2](https://huggingface.co/intfloat/e5-large-v2) | 1.34 | 1024| 512 | 62.25 | 44.49 | 86.03 | 56.61 | 50.56 | 82.05 | 30.19 | 75.24 | |
| 2624 | | [e5-base-v2](https://huggingface.co/intfloat/e5-base-v2) | 0.44 | 768 | 512 | 61.5 | 43.80 | 85.73 | 55.91 | 50.29 | 81.05 | 30.28 | 73.84 | |
| 2625 | | [**gte-small**](https://huggingface.co/thenlper/gte-small) | 0.07 | 384 | 512 | **61.36** | 44.89 | 83.54 | 57.7 | 49.46 | 82.07 | 30.42 | 72.31 | |
| 2626 | | [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings) | - | 1536 | 8192 | 60.99 | 45.9 | 84.89 | 56.32 | 49.25 | 80.97 | 30.8 | 70.93 | |
| 2627 | | [e5-small-v2](https://huggingface.co/intfloat/e5-base-v2) | 0.13 | 384 | 512 | 59.93 | 39.92 | 84.67 | 54.32 | 49.04 | 80.39 | 31.16 | 72.94 | |
| 2628 | | [sentence-t5-xxl](https://huggingface.co/sentence-transformers/sentence-t5-xxl) | 9.73 | 768 | 512 | 59.51 | 43.72 | 85.06 | 56.42 | 42.24 | 82.63 | 30.08 | 73.42 | |
| 2629 | | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) | 0.44 | 768 | 514 | 57.78 | 43.69 | 83.04 | 59.36 | 43.81 | 80.28 | 27.49 | 65.07 | |
| 2630 | | [sgpt-bloom-7b1-msmarco](https://huggingface.co/bigscience/sgpt-bloom-7b1-msmarco) | 28.27 | 4096 | 2048 | 57.59 | 38.93 | 81.9 | 55.65 | 48.22 | 77.74 | 33.6 | 66.19 | |
| 2631 | | [all-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2) | 0.13 | 384 | 512 | 56.53 | 41.81 | 82.41 | 58.44 | 42.69 | 79.8 | 27.9 | 63.21 | |
| 2632 | | [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | 0.09 | 384 | 512 | 56.26 | 42.35 | 82.37 | 58.04 | 41.95 | 78.9 | 30.81 | 63.05 | |
| 2633 | | [contriever-base-msmarco](https://huggingface.co/nthakur/contriever-base-msmarco) | 0.44 | 768 | 512 | 56.00 | 41.1 | 82.54 | 53.14 | 41.88 | 76.51 | 30.36 | 66.68 | |
| 2634 | | [sentence-t5-base](https://huggingface.co/sentence-transformers/sentence-t5-base) | 0.22 | 768 | 512 | 55.27 | 40.21 | 85.18 | 53.09 | 33.63 | 81.14 | 31.39 | 69.81 | |
| 2635 | |
| 2636 | |
| 2637 | ## Usage |
| 2638 | |
| 2639 | Code example |
| 2640 | |
| 2641 | ```python |
| 2642 | import torch.nn.functional as F |
| 2643 | from torch import Tensor |
| 2644 | from transformers import AutoTokenizer, AutoModel |
| 2645 | |
| 2646 | def average_pool(last_hidden_states: Tensor, |
| 2647 | attention_mask: Tensor) -> Tensor: |
| 2648 | last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0) |
| 2649 | return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None] |
| 2650 | |
| 2651 | input_texts = [ |
| 2652 | "what is the capital of China?", |
| 2653 | "how to implement quick sort in python?", |
| 2654 | "Beijing", |
| 2655 | "sorting algorithms" |
| 2656 | ] |
| 2657 | |
| 2658 | tokenizer = AutoTokenizer.from_pretrained("thenlper/gte-small") |
| 2659 | model = AutoModel.from_pretrained("thenlper/gte-small") |
| 2660 | |
| 2661 | # Tokenize the input texts |
| 2662 | batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt') |
| 2663 | |
| 2664 | outputs = model(**batch_dict) |
| 2665 | embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask']) |
| 2666 | |
| 2667 | # (Optionally) normalize embeddings |
| 2668 | embeddings = F.normalize(embeddings, p=2, dim=1) |
| 2669 | scores = (embeddings[:1] @ embeddings[1:].T) * 100 |
| 2670 | print(scores.tolist()) |
| 2671 | ``` |
| 2672 | |
| 2673 | Use with sentence-transformers: |
| 2674 | ```python |
| 2675 | from sentence_transformers import SentenceTransformer |
| 2676 | from sentence_transformers.util import cos_sim |
| 2677 | |
| 2678 | sentences = ['That is a happy person', 'That is a very happy person'] |
| 2679 | |
| 2680 | model = SentenceTransformer('thenlper/gte-large') |
| 2681 | embeddings = model.encode(sentences) |
| 2682 | print(cos_sim(embeddings[0], embeddings[1])) |
| 2683 | ``` |
| 2684 | |
| 2685 | ### Limitation |
| 2686 | |
| 2687 | This model exclusively caters to English texts, and any lengthy texts will be truncated to a maximum of 512 tokens. |
| 2688 | |
| 2689 | ### Citation |
| 2690 | |
| 2691 | If you find our paper or models helpful, please consider citing them as follows: |
| 2692 | |
| 2693 | ``` |
| 2694 | @article{li2023towards, |
| 2695 | title={Towards general text embeddings with multi-stage contrastive learning}, |
| 2696 | author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan}, |
| 2697 | journal={arXiv preprint arXiv:2308.03281}, |
| 2698 | year={2023} |
| 2699 | } |
| 2700 | ``` |
| 2701 | |