README.md
7.5 KB · 231 lines · markdown Raw
1 ---
2 language: en
3 license: cc-by-4.0
4 datasets:
5 - squad_v2
6 base_model: roberta-large
7 model-index:
8 - name: deepset/roberta-large-squad2
9 results:
10 - task:
11 type: question-answering
12 name: Question Answering
13 dataset:
14 name: squad_v2
15 type: squad_v2
16 config: squad_v2
17 split: validation
18 metrics:
19 - type: exact_match
20 value: 85.168
21 name: Exact Match
22 - type: f1
23 value: 88.349
24 name: F1
25 - task:
26 type: question-answering
27 name: Question Answering
28 dataset:
29 name: squad
30 type: squad
31 config: plain_text
32 split: validation
33 metrics:
34 - type: exact_match
35 value: 87.162
36 name: Exact Match
37 - type: f1
38 value: 93.603
39 name: F1
40 - task:
41 type: question-answering
42 name: Question Answering
43 dataset:
44 name: adversarial_qa
45 type: adversarial_qa
46 config: adversarialQA
47 split: validation
48 metrics:
49 - type: exact_match
50 value: 35.900
51 name: Exact Match
52 - type: f1
53 value: 48.923
54 name: F1
55 - task:
56 type: question-answering
57 name: Question Answering
58 dataset:
59 name: squad_adversarial
60 type: squad_adversarial
61 config: AddOneSent
62 split: validation
63 metrics:
64 - type: exact_match
65 value: 81.142
66 name: Exact Match
67 - type: f1
68 value: 87.099
69 name: F1
70 - task:
71 type: question-answering
72 name: Question Answering
73 dataset:
74 name: squadshifts amazon
75 type: squadshifts
76 config: amazon
77 split: test
78 metrics:
79 - type: exact_match
80 value: 72.453
81 name: Exact Match
82 - type: f1
83 value: 86.325
84 name: F1
85 - task:
86 type: question-answering
87 name: Question Answering
88 dataset:
89 name: squadshifts new_wiki
90 type: squadshifts
91 config: new_wiki
92 split: test
93 metrics:
94 - type: exact_match
95 value: 82.338
96 name: Exact Match
97 - type: f1
98 value: 91.974
99 name: F1
100 - task:
101 type: question-answering
102 name: Question Answering
103 dataset:
104 name: squadshifts nyt
105 type: squadshifts
106 config: nyt
107 split: test
108 metrics:
109 - type: exact_match
110 value: 84.352
111 name: Exact Match
112 - type: f1
113 value: 92.645
114 name: F1
115 - task:
116 type: question-answering
117 name: Question Answering
118 dataset:
119 name: squadshifts reddit
120 type: squadshifts
121 config: reddit
122 split: test
123 metrics:
124 - type: exact_match
125 value: 74.722
126 name: Exact Match
127 - type: f1
128 value: 86.860
129 name: F1
130 ---
131
132 # roberta-large for Extractive QA
133
134 This is the [roberta-large](https://huggingface.co/roberta-large) model, fine-tuned using the [SQuAD2.0](https://huggingface.co/datasets/squad_v2) dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Question Answering.
135
136
137 ## Overview
138 **Language model:** roberta-large
139 **Language:** English
140 **Downstream-task:** Extractive QA
141 **Training data:** SQuAD 2.0
142 **Eval data:** SQuAD 2.0
143 **Code:** See [an example extractive QA pipeline built with Haystack](https://haystack.deepset.ai/tutorials/34_extractive_qa_pipeline)
144 **Infrastructure**: 4x Tesla v100
145
146 ## Hyperparameters
147
148 ```
149 base_LM_model = "roberta-large"
150 ```
151
152 ## Using a distilled model instead
153 Please note that we have also released a distilled version of this model called [deepset/roberta-base-squad2-distilled](https://huggingface.co/deepset/roberta-base-squad2-distilled). The distilled model has a comparable prediction quality and runs at twice the speed of the large model.
154
155 ## Usage
156
157 ### In Haystack
158 Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents.
159 To load and run the model with [Haystack](https://github.com/deepset-ai/haystack/):
160 ```python
161 # After running pip install haystack-ai "transformers[torch,sentencepiece]"
162
163 from haystack import Document
164 from haystack.components.readers import ExtractiveReader
165
166 docs = [
167 Document(content="Python is a popular programming language"),
168 Document(content="python ist eine beliebte Programmiersprache"),
169 ]
170
171 reader = ExtractiveReader(model="deepset/roberta-large-squad2")
172 reader.warm_up()
173
174 question = "What is a popular programming language?"
175 result = reader.run(query=question, documents=docs)
176 # {'answers': [ExtractedAnswer(query='What is a popular programming language?', score=0.5740374326705933, data='python', document=Document(id=..., content: '...'), context=None, document_offset=ExtractedAnswer.Span(start=0, end=6),...)]}
177 ```
178 For a complete example with an extractive question answering pipeline that scales over many documents, check out the [corresponding Haystack tutorial](https://haystack.deepset.ai/tutorials/34_extractive_qa_pipeline).
179
180 ### In Transformers
181 ```python
182 from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline
183
184 model_name = "deepset/roberta-large-squad2"
185
186 # a) Get predictions
187 nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
188 QA_input = {
189 'question': 'Why is model conversion important?',
190 'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
191 }
192 res = nlp(QA_input)
193
194 # b) Load model & tokenizer
195 model = AutoModelForQuestionAnswering.from_pretrained(model_name)
196 tokenizer = AutoTokenizer.from_pretrained(model_name)
197 ```
198
199 ## Authors
200 **Branden Chan:** branden.chan@deepset.ai
201 **Timo Möller:** timo.moeller@deepset.ai
202 **Malte Pietsch:** malte.pietsch@deepset.ai
203 **Tanay Soni:** tanay.soni@deepset.ai
204
205 ## About us
206
207 <div class="grid lg:grid-cols-2 gap-x-4 gap-y-3">
208 <div class="w-full h-40 object-cover mb-2 rounded-lg flex items-center justify-center">
209 <img alt="" src="https://raw.githubusercontent.com/deepset-ai/.github/main/deepset-logo-colored.png" class="w-40"/>
210 </div>
211 <div class="w-full h-40 object-cover mb-2 rounded-lg flex items-center justify-center">
212 <img alt="" src="https://raw.githubusercontent.com/deepset-ai/.github/main/haystack-logo-colored.png" class="w-40"/>
213 </div>
214 </div>
215
216 [deepset](http://deepset.ai/) is the company behind the production-ready open-source AI framework [Haystack](https://haystack.deepset.ai/).
217
218 Some of our other work:
219 - [Distilled roberta-base-squad2 (aka "tinyroberta-squad2")](https://huggingface.co/deepset/tinyroberta-squad2)
220 - [German BERT](https://deepset.ai/german-bert), [GermanQuAD and GermanDPR](https://deepset.ai/germanquad), [German embedding model](https://huggingface.co/mixedbread-ai/deepset-mxbai-embed-de-large-v1)
221 - [deepset Cloud](https://www.deepset.ai/deepset-cloud-product), [deepset Studio](https://www.deepset.ai/deepset-studio)
222
223 ## Get in touch and join the Haystack community
224
225 <p>For more info on Haystack, visit our <strong><a href="https://github.com/deepset-ai/haystack">GitHub</a></strong> repo and <strong><a href="https://docs.haystack.deepset.ai">Documentation</a></strong>.
226
227 We also have a <strong><a class="h-7" href="https://haystack.deepset.ai/community">Discord community open to everyone!</a></strong></p>
228
229 [Twitter](https://twitter.com/Haystack_AI) | [LinkedIn](https://www.linkedin.com/company/deepset-ai/) | [Discord](https://haystack.deepset.ai/community) | [GitHub Discussions](https://github.com/deepset-ai/haystack/discussions) | [Website](https://haystack.deepset.ai/) | [YouTube](https://www.youtube.com/@deepset_ai)
230
231 By the way: [we're hiring!](http://www.deepset.ai/jobs)