README.md
14.5 KB · 259 lines · markdown Raw
1 ---
2 license: mit
3 license_link: https://huggingface.co/microsoft/Florence-2-base/resolve/main/LICENSE
4 pipeline_tag: image-text-to-text
5 tags:
6 - vision
7 ---
8
9 # Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
10
11 ## Model Summary
12
13 This Hub repository contains a HuggingFace's `transformers` implementation of Florence-2 model from Microsoft.
14
15 Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages our FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.
16
17 Resources and Technical Documentation:
18 + [Florence-2 technical report](https://arxiv.org/abs/2311.06242).
19 + [Jupyter Notebook for inference and visualization of Florence-2-large model](https://huggingface.co/microsoft/Florence-2-large/blob/main/sample_inference.ipynb)
20
21 | Model | Model size | Model Description |
22 | ------- | ------------- | ------------- |
23 | Florence-2-base[[HF]](https://huggingface.co/microsoft/Florence-2-base) | 0.23B | Pretrained model with FLD-5B
24 | Florence-2-large[[HF]](https://huggingface.co/microsoft/Florence-2-large) | 0.77B | Pretrained model with FLD-5B
25 | Florence-2-base-ft[[HF]](https://huggingface.co/microsoft/Florence-2-base-ft) | 0.23B | Finetuned model on a colletion of downstream tasks
26 | Florence-2-large-ft[[HF]](https://huggingface.co/microsoft/Florence-2-large-ft) | 0.77B | Finetuned model on a colletion of downstream tasks
27
28 ## How to Get Started with the Model
29
30 Use the code below to get started with the model. All models are trained with float16.
31
32 ```python
33 import requests
34
35 from PIL import Image
36 from transformers import AutoProcessor, AutoModelForCausalLM
37
38 device = "cuda:0" if torch.cuda.is_available() else "cpu"
39 torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
40
41 model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-base", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
42 processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base", trust_remote_code=True)
43
44 prompt = "<OD>"
45
46 url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
47 image = Image.open(requests.get(url, stream=True).raw)
48
49 inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
50
51 generated_ids = model.generate(
52 input_ids=inputs["input_ids"],
53 pixel_values=inputs["pixel_values"],
54 max_new_tokens=1024,
55 do_sample=False,
56 num_beams=3,
57 )
58 generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
59
60 parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))
61
62 print(parsed_answer)
63
64 ```
65
66
67 ## Tasks
68
69 This model is capable of performing different tasks through changing the prompts.
70
71 First, let's define a function to run a prompt.
72
73 <details>
74 <summary> Click to expand </summary>
75
76 ```python
77 import requests
78
79 from PIL import Image
80 from transformers import AutoProcessor, AutoModelForCausalLM
81
82 device = "cuda:0" if torch.cuda.is_available() else "cpu"
83 torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
84
85 model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-base", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
86 processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base", trust_remote_code=True)
87
88 url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
89 image = Image.open(requests.get(url, stream=True).raw)
90
91 def run_example(task_prompt, text_input=None):
92 if text_input is None:
93 prompt = task_prompt
94 else:
95 prompt = task_prompt + text_input
96 inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
97 generated_ids = model.generate(
98 input_ids=inputs["input_ids"],
99 pixel_values=inputs["pixel_values"],
100 max_new_tokens=1024,
101 num_beams=3
102 )
103 generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
104
105 parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))
106
107 print(parsed_answer)
108 ```
109 </details>
110
111 Here are the tasks `Florence-2` could perform:
112
113 <details>
114 <summary> Click to expand </summary>
115
116
117
118 ### Caption
119 ```python
120 prompt = "<CAPTION>"
121 run_example(prompt)
122 ```
123
124 ### Detailed Caption
125 ```python
126 prompt = "<DETAILED_CAPTION>"
127 run_example(prompt)
128 ```
129
130 ### More Detailed Caption
131 ```python
132 prompt = "<MORE_DETAILED_CAPTION>"
133 run_example(prompt)
134 ```
135
136 ### Caption to Phrase Grounding
137 caption to phrase grounding task requires additional text input, i.e. caption.
138
139 Caption to phrase grounding results format:
140 {'\<CAPTION_TO_PHRASE_GROUNDING>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': ['', '', ...]}}
141 ```python
142 task_prompt = "<CAPTION_TO_PHRASE_GROUNDING>"
143 results = run_example(task_prompt, text_input="A green car parked in front of a yellow building.")
144 ```
145
146 ### Object Detection
147
148 OD results format:
149 {'\<OD>': {'bboxes': [[x1, y1, x2, y2], ...],
150 'labels': ['label1', 'label2', ...]} }
151
152 ```python
153 prompt = "<OD>"
154 run_example(prompt)
155 ```
156
157 ### Dense Region Caption
158 Dense region caption results format:
159 {'\<DENSE_REGION_CAPTION>' : {'bboxes': [[x1, y1, x2, y2], ...],
160 'labels': ['label1', 'label2', ...]} }
161 ```python
162 prompt = "<DENSE_REGION_CAPTION>"
163 run_example(prompt)
164 ```
165
166 ### Region proposal
167 Dense region caption results format:
168 {'\<REGION_PROPOSAL>': {'bboxes': [[x1, y1, x2, y2], ...],
169 'labels': ['', '', ...]}}
170 ```python
171 prompt = "<REGION_PROPOSAL>"
172 run_example(prompt)
173 ```
174
175 ### OCR
176
177 ```python
178 prompt = "<OCR>"
179 run_example(prompt)
180 ```
181
182 ### OCR with Region
183 OCR with region output format:
184 {'\<OCR_WITH_REGION>': {'quad_boxes': [[x1, y1, x2, y2, x3, y3, x4, y4], ...], 'labels': ['text1', ...]}}
185 ```python
186 prompt = "<OCR_WITH_REGION>"
187 run_example(prompt)
188 ```
189
190 for More detailed examples, please refer to [notebook](https://huggingface.co/microsoft/Florence-2-large/blob/main/sample_inference.ipynb)
191 </details>
192
193 # Benchmarks
194
195 ## Florence-2 Zero-shot performance
196
197 The following table presents the zero-shot performance of generalist vision foundation models on image captioning and object detection evaluation tasks. These models have not been exposed to the training data of the evaluation tasks during their training phase.
198
199 | Method | #params | COCO Cap. test CIDEr | NoCaps val CIDEr | TextCaps val CIDEr | COCO Det. val2017 mAP |
200 |--------|---------|----------------------|------------------|--------------------|-----------------------|
201 | Flamingo | 80B | 84.3 | - | - | - |
202 | Florence-2-base| 0.23B | 133.0 | 118.7 | 70.1 | 34.7 |
203 | Florence-2-large| 0.77B | 135.6 | 120.8 | 72.8 | 37.5 |
204
205
206 The following table continues the comparison with performance on other vision-language evaluation tasks.
207
208 | Method | Flickr30k test R@1 | Refcoco val Accuracy | Refcoco test-A Accuracy | Refcoco test-B Accuracy | Refcoco+ val Accuracy | Refcoco+ test-A Accuracy | Refcoco+ test-B Accuracy | Refcocog val Accuracy | Refcocog test Accuracy | Refcoco RES val mIoU |
209 |--------|----------------------|----------------------|-------------------------|-------------------------|-----------------------|--------------------------|--------------------------|-----------------------|------------------------|----------------------|
210 | Kosmos-2 | 78.7 | 52.3 | 57.4 | 47.3 | 45.5 | 50.7 | 42.2 | 60.6 | 61.7 | - |
211 | Florence-2-base | 83.6 | 53.9 | 58.4 | 49.7 | 51.5 | 56.4 | 47.9 | 66.3 | 65.1 | 34.6 |
212 | Florence-2-large | 84.4 | 56.3 | 61.6 | 51.4 | 53.6 | 57.9 | 49.9 | 68.0 | 67.0 | 35.8 |
213
214
215
216 ## Florence-2 finetuned performance
217
218 We finetune Florence-2 models with a collection of downstream tasks, resulting two generalist models *Florence-2-base-ft* and *Florence-2-large-ft* that can conduct a wide range of downstream tasks.
219
220 The table below compares the performance of specialist and generalist models on various captioning and Visual Question Answering (VQA) tasks. Specialist models are fine-tuned specifically for each task, whereas generalist models are fine-tuned in a task-agnostic manner across all tasks. The symbol "▲" indicates the usage of external OCR as input.
221
222 | Method | # Params | COCO Caption Karpathy test CIDEr | NoCaps val CIDEr | TextCaps val CIDEr | VQAv2 test-dev Acc | TextVQA test-dev Acc | VizWiz VQA test-dev Acc |
223 |----------------|----------|-----------------------------------|------------------|--------------------|--------------------|----------------------|-------------------------|
224 | **Specialist Models** | | | | | | | |
225 | CoCa | 2.1B | 143.6 | 122.4 | - | 82.3 | - | - |
226 | BLIP-2 | 7.8B | 144.5 | 121.6 | - | 82.2 | - | - |
227 | GIT2 | 5.1B | 145.0 | 126.9 | 148.6 | 81.7 | 67.3 | 71.0 |
228 | Flamingo | 80B | 138.1 | - | - | 82.0 | 54.1 | 65.7 |
229 | PaLI | 17B | 149.1 | 127.0 | 160.0▲ | 84.3 | 58.8 / 73.1▲ | 71.6 / 74.4▲ |
230 | PaLI-X | 55B | 149.2 | 126.3 | 147.0 / 163.7▲ | 86.0 | 71.4 / 80.8▲ | 70.9 / 74.6▲ |
231 | **Generalist Models** | | | | | | | |
232 | Unified-IO | 2.9B | - | 100.0 | - | 77.9 | - | 57.4 |
233 | Florence-2-base-ft | 0.23B | 140.0 | 116.7 | 143.9 | 79.7 | 63.6 | 63.6 |
234 | Florence-2-large-ft | 0.77B | 143.3 | 124.9 | 151.1 | 81.7 | 73.5 | 72.6 |
235
236
237 | Method | # Params | COCO Det. val2017 mAP | Flickr30k test R@1 | RefCOCO val Accuracy | RefCOCO test-A Accuracy | RefCOCO test-B Accuracy | RefCOCO+ val Accuracy | RefCOCO+ test-A Accuracy | RefCOCO+ test-B Accuracy | RefCOCOg val Accuracy | RefCOCOg test Accuracy | RefCOCO RES val mIoU |
238 |----------------------|----------|-----------------------|--------------------|----------------------|-------------------------|-------------------------|------------------------|---------------------------|---------------------------|------------------------|-----------------------|------------------------|
239 | **Specialist Models** | | | | | | | | | | | | |
240 | SeqTR | - | - | - | 83.7 | 86.5 | 81.2 | 71.5 | 76.3 | 64.9 | 74.9 | 74.2 | - |
241 | PolyFormer | - | - | - | 90.4 | 92.9 | 87.2 | 85.0 | 89.8 | 78.0 | 85.8 | 85.9 | 76.9 |
242 | UNINEXT | 0.74B | 60.6 | - | 92.6 | 94.3 | 91.5 | 85.2 | 89.6 | 79.8 | 88.7 | 89.4 | - |
243 | Ferret | 13B | - | - | 89.5 | 92.4 | 84.4 | 82.8 | 88.1 | 75.2 | 85.8 | 86.3 | - |
244 | **Generalist Models** | | | | | | | | | | | | |
245 | UniTAB | - | - | - | 88.6 | 91.1 | 83.8 | 81.0 | 85.4 | 71.6 | 84.6 | 84.7 | - |
246 | Florence-2-base-ft | 0.23B | 41.4 | 84.0 | 92.6 | 94.8 | 91.5 | 86.8 | 91.7 | 82.2 | 89.8 | 82.2 | 78.0 |
247 | Florence-2-large-ft| 0.77B | 43.4 | 85.2 | 93.4 | 95.3 | 92.0 | 88.3 | 92.9 | 83.6 | 91.2 | 91.7 | 80.5 |
248
249
250 ## BibTex and citation info
251
252 ```
253 @article{xiao2023florence,
254 title={Florence-2: Advancing a unified representation for a variety of vision tasks},
255 author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
256 journal={arXiv preprint arXiv:2311.06242},
257 year={2023}
258 }
259 ```