README.md
15.4 KB · 287 lines · markdown Raw
1 ---
2 pipeline_tag: text-ranking
3 tags:
4 - transformers
5 - reranker
6 - cross-encoder
7 - transformers.js
8 - sentence-transformers
9 language:
10 - multilingual
11 inference: false
12 license: cc-by-nc-4.0
13 library_name: transformers
14 ---
15
16 <br><br>
17
18 <p align="center">
19 <img src="https://huggingface.co/datasets/jinaai/documentation-images/resolve/main/logo.webp" alt="Jina AI: Your Search Foundation, Supercharged!" width="150px">
20 </p>
21
22 <p align="center">
23 <b>Trained by <a href="https://jina.ai/"><b>Jina AI</b></a>.</b>
24 </p>
25
26 # jina-reranker-v2-base-multilingual
27
28 ## Intended Usage & Model Info
29
30 The **Jina Reranker v2** (`jina-reranker-v2-base-multilingual`) is a transformer-based model that has been fine-tuned for text reranking task, which is a crucial component in many information retrieval systems. It is a cross-encoder model that takes a query and a document pair as input and outputs a score indicating the relevance of the document to the query. The model is trained on a large dataset of query-document pairs and is capable of reranking documents in multiple languages with high accuracy.
31
32 Compared with the state-of-the-art reranker models, including the previous released `jina-reranker-v1-base-en`, the **Jina Reranker v2** model has demonstrated competitiveness across a series of benchmarks targeting for text retrieval, multilingual capability, function-calling-aware and text-to-SQL-aware reranking, and code retrieval tasks.
33
34 The `jina-reranker-v2-base-multilingual` model is capable of handling long texts with a context length of up to `1024` tokens, enabling the processing of extensive inputs. To enable the model to handle long texts that exceed 1024 tokens, the model uses a sliding window approach to chunk the input text into smaller pieces and rerank each chunk separately.
35
36 The model is also equipped with a flash attention mechanism, which significantly improves the model's performance.
37
38
39 # Usage
40
41 _This model repository is licenced for research and evaluation purposes under CC-BY-NC-4.0. For commercial usage, please refer to Jina AI's APIs, AWS Sagemaker or Azure Marketplace offerings. Please [contact us](https://jina.ai/contact-sales) for any further clarifications._
42 1. The easiest way to use `jina-reranker-v2-base-multilingual` is to call Jina AI's [Reranker API](https://jina.ai/reranker/).
43
44 ```bash
45 curl https://api.jina.ai/v1/rerank \
46 -H "Content-Type: application/json" \
47 -H "Authorization: Bearer YOUR_API_KEY" \
48 -d '{
49 "model": "jina-reranker-v2-base-multilingual",
50 "query": "Organic skincare products for sensitive skin",
51 "documents": [
52 "Organic skincare for sensitive skin with aloe vera and chamomile.",
53 "New makeup trends focus on bold colors and innovative techniques",
54 "Bio-Hautpflege für empfindliche Haut mit Aloe Vera und Kamille",
55 "Neue Make-up-Trends setzen auf kräftige Farben und innovative Techniken",
56 "Cuidado de la piel orgánico para piel sensible con aloe vera y manzanilla",
57 "Las nuevas tendencias de maquillaje se centran en colores vivos y técnicas innovadoras",
58 "针对敏感肌专门设计的天然有机护肤产品",
59 "新的化妆趋势注重鲜艳的颜色和创新的技巧",
60 "敏感肌のために特別に設計された天然有機スキンケア製品",
61 "新しいメイクのトレンドは鮮やかな色と革新的な技術に焦点を当てています"
62 ],
63 "top_n": 3
64 }'
65 ```
66
67 2. You can also use the `transformers` library to interact with the model programmatically.
68
69 Before you start, install the `transformers` and `einops` libraries:
70
71 ```bash
72 pip install transformers einops
73 ```
74
75 And then:
76 ```python
77 from transformers import AutoModelForSequenceClassification
78
79 model = AutoModelForSequenceClassification.from_pretrained(
80 'jinaai/jina-reranker-v2-base-multilingual',
81 torch_dtype="auto",
82 trust_remote_code=True,
83 )
84
85 model.to('cuda') # or 'cpu' if no GPU is available
86 model.eval()
87
88 # Example query and documents
89 query = "Organic skincare products for sensitive skin"
90 documents = [
91 "Organic skincare for sensitive skin with aloe vera and chamomile.",
92 "New makeup trends focus on bold colors and innovative techniques",
93 "Bio-Hautpflege für empfindliche Haut mit Aloe Vera und Kamille",
94 "Neue Make-up-Trends setzen auf kräftige Farben und innovative Techniken",
95 "Cuidado de la piel orgánico para piel sensible con aloe vera y manzanilla",
96 "Las nuevas tendencias de maquillaje se centran en colores vivos y técnicas innovadoras",
97 "针对敏感肌专门设计的天然有机护肤产品",
98 "新的化妆趋势注重鲜艳的颜色和创新的技巧",
99 "敏感肌のために特別に設計された天然有機スキンケア製品",
100 "新しいメイクのトレンドは鮮やかな色と革新的な技術に焦点を当てています",
101 ]
102
103 # construct sentence pairs
104 sentence_pairs = [[query, doc] for doc in documents]
105
106 scores = model.compute_score(sentence_pairs, max_length=1024)
107 ```
108
109 The scores will be a list of floats, where each float represents the relevance score of the corresponding document to the query. Higher scores indicate higher relevance.
110 For instance the returning scores in this case will be:
111 ```bash
112 [0.8311430811882019, 0.09401018172502518,
113 0.6334102749824524, 0.08269733935594559,
114 0.7620701193809509, 0.09947021305561066,
115 0.9263036847114563, 0.05834583938121796,
116 0.8418256044387817, 0.11124119907617569]
117 ```
118
119 The model gives high relevance scores to the documents that are most relevant to the query regardless of the language of the document.
120
121 Note that by default, the `jina-reranker-v2-base-multilingual` model uses [flash attention](https://github.com/Dao-AILab/flash-attention), which requires certain types of GPU hardware to run.
122 If you encounter any issues, you can try call `AutoModelForSequenceClassification.from_pretrained()` with `use_flash_attn=False`.
123 This will use the standard attention mechanism instead of flash attention.
124
125 If you want to use flash attention for fast inference, you need to install the following packages:
126 ```bash
127 pip install ninja # required for flash attention
128 pip install flash-attn --no-build-isolation
129 ```
130 Enjoy the 3x-6x speedup with flash attention! ⚡️⚡️⚡️
131
132
133 3. You can also use the `transformers.js` library to run the model directly in JavaScript (in-browser, Node.js, Deno, etc.)!
134
135 If you haven't already, you can install the [Transformers.js](https://huggingface.co/docs/transformers.js) JavaScript library (v3) using:
136 ```bash
137 npm i xenova/transformers.js#v3
138 ```
139
140 Then, you can use the following code to interact with the model:
141 ```js
142 import { AutoTokenizer, XLMRobertaModel } from '@xenova/transformers';
143
144 const model_id = 'jinaai/jina-reranker-v2-base-multilingual';
145 const model = await XLMRobertaModel.from_pretrained(model_id, { dtype: 'fp32' });
146 const tokenizer = await AutoTokenizer.from_pretrained(model_id);
147
148 /**
149 * Performs ranking with the CrossEncoder on the given query and documents. Returns a sorted list with the document indices and scores.
150 * @param {string} query A single query
151 * @param {string[]} documents A list of documents
152 * @param {Object} options Options for ranking
153 * @param {number} [options.top_k=undefined] Return the top-k documents. If undefined, all documents are returned.
154 * @param {number} [options.return_documents=false] If true, also returns the documents. If false, only returns the indices and scores.
155 */
156 async function rank(query, documents, {
157 top_k = undefined,
158 return_documents = false,
159 } = {}) {
160 const inputs = tokenizer(
161 new Array(documents.length).fill(query),
162 { text_pair: documents, padding: true, truncation: true }
163 )
164 const { logits } = await model(inputs);
165 return logits.sigmoid().tolist()
166 .map(([score], i) => ({
167 corpus_id: i,
168 score,
169 ...(return_documents ? { text: documents[i] } : {})
170 })).sort((a, b) => b.score - a.score).slice(0, top_k);
171 }
172
173 // Example usage:
174 const query = "Organic skincare products for sensitive skin"
175 const documents = [
176 "Organic skincare for sensitive skin with aloe vera and chamomile.",
177 "New makeup trends focus on bold colors and innovative techniques",
178 "Bio-Hautpflege für empfindliche Haut mit Aloe Vera und Kamille",
179 "Neue Make-up-Trends setzen auf kräftige Farben und innovative Techniken",
180 "Cuidado de la piel orgánico para piel sensible con aloe vera y manzanilla",
181 "Las nuevas tendencias de maquillaje se centran en colores vivos y técnicas innovadoras",
182 "针对敏感肌专门设计的天然有机护肤产品",
183 "新的化妆趋势注重鲜艳的颜色和创新的技巧",
184 "敏感肌のために特別に設計された天然有機スキンケア製品",
185 "新しいメイクのトレンドは鮮やかな色と革新的な技術に焦点を当てています",
186 ]
187
188 const results = await rank(query, documents, { return_documents: true, top_k: 3 });
189 console.log(results);
190 ```
191
192
193 That's it! You can now use the `jina-reranker-v2-base-multilingual` model in your projects.
194
195
196 In addition to the `compute_score()` function, the `jina-reranker-v2-base-multilingual` model also provides a `model.rerank()` function that can be used to rerank documents based on a query. You can use it as follows:
197
198 ```python
199 result = model.rerank(
200 query,
201 documents,
202 max_query_length=512,
203 max_length=1024,
204 top_n=3
205 )
206 ```
207
208 Inside the `result` object, you will find the reranked documents along with their scores. You can use this information to further process the documents as needed.
209
210 The `rerank()` function will automatically chunk the input documents into smaller pieces if they exceed the model's maximum input length. This allows you to rerank long documents without running into memory issues.
211 Specifically, the `rerank()` function will split the documents into chunks of size `max_length` and rerank each chunk separately. The scores from all the chunks are then combined to produce the final reranking results. You can control the query length and document length in each chunk by setting the `max_query_length` and `max_length` parameters. The `rerank()` function also supports the `overlap` parameter (default is `80`) which determines how much overlap there is between adjacent chunks. This can be useful when reranking long documents to ensure that the model has enough context to make accurate predictions.
212
213 3. Alternatively, `jina-reranker-v2-base-multilingual` has been integrated with `CrossEncoder` from the `sentence-transformers` library.
214
215 Before you start, install the `sentence-transformers` libraries:
216
217 ```bash
218 pip install sentence-transformers
219 ```
220
221 The [`CrossEncoder`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html) class supports a [`predict`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.predict) method to get query-document relevance scores, and a [`rank`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.rank) method to rank all documents given your query.
222
223 ```python
224 from sentence_transformers import CrossEncoder
225
226 model = CrossEncoder(
227 "jinaai/jina-reranker-v2-base-multilingual",
228 automodel_args={"torch_dtype": "auto"},
229 trust_remote_code=True,
230 )
231
232 # Example query and documents
233 query = "Organic skincare products for sensitive skin"
234 documents = [
235 "Organic skincare for sensitive skin with aloe vera and chamomile.",
236 "New makeup trends focus on bold colors and innovative techniques",
237 "Bio-Hautpflege für empfindliche Haut mit Aloe Vera und Kamille",
238 "Neue Make-up-Trends setzen auf kräftige Farben und innovative Techniken",
239 "Cuidado de la piel orgánico para piel sensible con aloe vera y manzanilla",
240 "Las nuevas tendencias de maquillaje se centran en colores vivos y técnicas innovadoras",
241 "针对敏感肌专门设计的天然有机护肤产品",
242 "新的化妆趋势注重鲜艳的颜色和创新的技巧",
243 "敏感肌のために特別に設計された天然有機スキンケア製品",
244 "新しいメイクのトレンドは鮮やかな色と革新的な技術に焦点を当てています",
245 ]
246
247 # construct sentence pairs
248 sentence_pairs = [[query, doc] for doc in documents]
249
250 scores = model.predict(sentence_pairs, convert_to_tensor=True).tolist()
251 """
252 [0.828125, 0.0927734375, 0.6328125, 0.08251953125, 0.76171875, 0.099609375, 0.92578125, 0.058349609375, 0.84375, 0.111328125]
253 """
254
255 rankings = model.rank(query, documents, return_documents=True, convert_to_tensor=True)
256 print(f"Query: {query}")
257 for ranking in rankings:
258 print(f"ID: {ranking['corpus_id']}, Score: {ranking['score']:.4f}, Text: {ranking['text']}")
259 """
260 Query: Organic skincare products for sensitive skin
261 ID: 6, Score: 0.9258, Text: 针对敏感肌专门设计的天然有机护肤产品
262 ID: 8, Score: 0.8438, Text: 敏感肌のために特別に設計された天然有機スキンケア製品
263 ID: 0, Score: 0.8281, Text: Organic skincare for sensitive skin with aloe vera and chamomile.
264 ID: 4, Score: 0.7617, Text: Cuidado de la piel orgánico para piel sensible con aloe vera y manzanilla
265 ID: 2, Score: 0.6328, Text: Bio-Hautpflege für empfindliche Haut mit Aloe Vera und Kamille
266 ID: 9, Score: 0.1113, Text: 新しいメイクのトレンドは鮮やかな色と革新的な技術に焦点を当てています
267 ID: 5, Score: 0.0996, Text: Las nuevas tendencias de maquillaje se centran en colores vivos y técnicas innovadoras
268 ID: 1, Score: 0.0928, Text: New makeup trends focus on bold colors and innovative techniques
269 ID: 3, Score: 0.0825, Text: Neue Make-up-Trends setzen auf kräftige Farben und innovative Techniken
270 ID: 7, Score: 0.0583, Text: 新的化妆趋势注重鲜艳的颜色和创新的技巧
271 """
272 ```
273
274 # Evaluation
275
276 We evaluated Jina Reranker v2 on multiple benchmarks to ensure top-tier performance and search relevance.
277
278 | Model Name | Model Size | MKQA(nDCG@10, 26 langs) | BEIR(nDCG@10, 17 datasets) | MLDR(recall@10, 13 langs) | CodeSearchNet (MRR@10, 3 tasks) | AirBench (nDCG@10, zh/en) | ToolBench (recall@3, 3 tasks) | TableSearch (recall@3) |
279 | :-----------------------------: | :----------: | ------------------------- | ---------------------------- | --------------------------- | --------------------------------- | --------------------------- | ------------------------------- | ------------------------ |
280 | jina-reranker-v2-multilingual | 278M | 54.83 | 53.17 | 68.95 | 71.36 | 61.33 | 77.75 | 93.31 |
281 | bge-reranker-v2-m3 | 568M | 54.17 | 53.65 | 59.73 | 62.86 | 61.28 | 78.46 | 74.86 |
282 | mmarco-mMiniLMv2-L12-H384-v1 | 118M | 53.37 | 45.40 | 28.91 | 51.78 | 56.46 | 58.39 | 53.60 |
283 | jina-reranker-v1-base-en | 137M | - | 52.45 | - | - | - | 74.13 | 72.89 |
284
285 Note:
286 - NDCG@10 and MRR@10 measure ranking quality, with higher scores indicating better search results
287 - recall@3 measures the proportion of relevant documents retrieved, with higher scores indicating better search results