README.md
10.2 KB · 291 lines · markdown Raw
1 ---
2 license: apache-2.0
3 pipeline_tag: image-text-to-text
4 library_name: transformers
5 base_model:
6 - Qwen/Qwen3-VL-8B-Instruct
7 base_model_relation: quantized
8 ---
9 <a href="https://chat.qwenlm.ai/" target="_blank" style="margin: 2px;">
10 <img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
11 </a>
12
13 # Qwen3-VL-8B-Instruct-FP8
14
15 > This repository contains an FP8 quantized version of the [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) model. The quantization method is fine-grained fp8 quantization with block size of 128, and its performance metrics are nearly identical to those of the original BF16 model. Enjoy!
16
17
18 Meet Qwen3-VL — the most powerful vision-language model in the Qwen series to date.
19
20 This generation delivers comprehensive upgrades across the board: superior text understanding & generation, deeper visual perception & reasoning, extended context length, enhanced spatial and video dynamics comprehension, and stronger agent interaction capabilities.
21
22 Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning‑enhanced Thinking editions for flexible, on‑demand deployment.
23
24
25 #### Key Enhancements:
26
27 * **Visual Agent**: Operates PC/mobile GUIs—recognizes elements, understands functions, invokes tools, completes tasks.
28
29 * **Visual Coding Boost**: Generates Draw.io/HTML/CSS/JS from images/videos.
30
31 * **Advanced Spatial Perception**: Judges object positions, viewpoints, and occlusions; provides stronger 2D grounding and enables 3D grounding for spatial reasoning and embodied AI.
32
33 * **Long Context & Video Understanding**: Native 256K context, expandable to 1M; handles books and hours-long video with full recall and second-level indexing.
34
35 * **Enhanced Multimodal Reasoning**: Excels in STEM/Math—causal analysis and logical, evidence-based answers.
36
37 * **Upgraded Visual Recognition**: Broader, higher-quality pretraining is able to “recognize everything”—celebrities, anime, products, landmarks, flora/fauna, etc.
38
39 * **Expanded OCR**: Supports 32 languages (up from 19); robust in low light, blur, and tilt; better with rare/ancient characters and jargon; improved long-document structure parsing.
40
41 * **Text Understanding on par with pure LLMs**: Seamless text–vision fusion for lossless, unified comprehension.
42
43
44 #### Model Architecture Updates:
45
46 <p align="center">
47 <img src="https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3-VL/qwen3vl_arc.jpg" width="80%"/>
48 <p>
49
50
51 1. **Interleaved-MRoPE**: Full‑frequency allocation over time, width, and height via robust positional embeddings, enhancing long‑horizon video reasoning.
52
53 2. **DeepStack**: Fuses multi‑level ViT features to capture fine‑grained details and sharpen image–text alignment.
54
55 3. **Text–Timestamp Alignment:** Moves beyond T‑RoPE to precise, timestamp‑grounded event localization for stronger video temporal modeling.
56
57
58 This is the weight repository for Qwen3-VL-8B-Instruct-FP8.
59
60
61 ---
62
63 ## Model Performance
64
65 **Multimodal performance**
66
67 ![](https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3-VL/qwen3vl_4b_8b_vl_instruct.jpg)
68
69 **Pure text performance**
70 ![](https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3-VL/qwen3vl_4b_8b_text_instruct.jpg)
71
72
73 ## Quickstart
74
75 Currently, 🤗 Transformers does not support loading these weights directly. Stay tuned!
76
77 We recommend deploying the model using vLLM or SGLang, with example launch commands provided below. For details on the runtime environment and deployment, please refer to this [link](https://github.com/QwenLM/Qwen3-VL?tab=readme-ov-file#deployment).
78
79 ### vLLM Inference
80
81 Here we provide a code snippet demonstrating how to use vLLM to run inference with Qwen3-VL locally. For more details on efficient deployment with vLLM, please refer to the [community deployment guide](https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-VL.html).
82
83 ```python
84 # -*- coding: utf-8 -*-
85 import torch
86 from qwen_vl_utils import process_vision_info
87 from transformers import AutoProcessor
88 from vllm import LLM, SamplingParams
89
90 import os
91 os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn'
92
93 def prepare_inputs_for_vllm(messages, processor):
94 text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
95 # qwen_vl_utils 0.0.14+ reqired
96 image_inputs, video_inputs, video_kwargs = process_vision_info(
97 messages,
98 image_patch_size=processor.image_processor.patch_size,
99 return_video_kwargs=True,
100 return_video_metadata=True
101 )
102 print(f"video_kwargs: {video_kwargs}")
103
104 mm_data = {}
105 if image_inputs is not None:
106 mm_data['image'] = image_inputs
107 if video_inputs is not None:
108 mm_data['video'] = video_inputs
109
110 return {
111 'prompt': text,
112 'multi_modal_data': mm_data,
113 'mm_processor_kwargs': video_kwargs
114 }
115
116
117 if __name__ == '__main__':
118 # messages = [
119 # {
120 # "role": "user",
121 # "content": [
122 # {
123 # "type": "video",
124 # "video": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-VL/space_woaudio.mp4",
125 # },
126 # {"type": "text", "text": "这段视频有多长"},
127 # ],
128 # }
129 # ]
130
131 messages = [
132 {
133 "role": "user",
134 "content": [
135 {
136 "type": "image",
137 "image": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3-VL/receipt.png",
138 },
139 {"type": "text", "text": "Read all the text in the image."},
140 ],
141 }
142 ]
143
144 # TODO: change to your own checkpoint path
145 checkpoint_path = "Qwen/Qwen3-VL-8B-Instruct-FP8"
146 processor = AutoProcessor.from_pretrained(checkpoint_path)
147 inputs = [prepare_inputs_for_vllm(message, processor) for message in [messages]]
148
149 llm = LLM(
150 model=checkpoint_path,
151 trust_remote_code=True,
152 gpu_memory_utilization=0.70,
153 enforce_eager=False,
154 tensor_parallel_size=torch.cuda.device_count(),
155 seed=0
156 )
157
158 sampling_params = SamplingParams(
159 temperature=0,
160 max_tokens=1024,
161 top_k=-1,
162 stop_token_ids=[],
163 )
164
165 for i, input_ in enumerate(inputs):
166 print()
167 print('=' * 40)
168 print(f"Inputs[{i}]: {input_['prompt']=!r}")
169 print('\n' + '>' * 40)
170
171 outputs = llm.generate(inputs, sampling_params=sampling_params)
172 for i, output in enumerate(outputs):
173 generated_text = output.outputs[0].text
174 print()
175 print('=' * 40)
176 print(f"Generated text: {generated_text!r}")
177 ```
178
179 ### SGLang Inference
180
181 Here we provide a code snippet demonstrating how to use SGLang to run inference with Qwen3-VL locally.
182
183 ```python
184 import time
185 from PIL import Image
186 from sglang import Engine
187 from qwen_vl_utils import process_vision_info
188 from transformers import AutoProcessor, AutoConfig
189
190 if __name__ == "__main__":
191 # TODO: change to your own checkpoint path
192 checkpoint_path = "Qwen/Qwen3-VL-8B-Instruct-FP8"
193 processor = AutoProcessor.from_pretrained(checkpoint_path)
194
195 messages = [
196 {
197 "role": "user",
198 "content": [
199 {
200 "type": "image",
201 "image": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3-VL/receipt.png",
202 },
203 {"type": "text", "text": "Read all the text in the image."},
204 ],
205 }
206 ]
207
208 text = processor.apply_chat_template(
209 messages,
210 tokenize=False,
211 add_generation_prompt=True
212 )
213
214 image_inputs, _ = process_vision_info(messages, image_patch_size=processor.image_processor.patch_size)
215
216 llm = Engine(
217 model_path=checkpoint_path,
218 enable_multimodal=True,
219 mem_fraction_static=0.8,
220 tp_size=torch.cuda.device_count(),
221 attention_backend="fa3"
222 )
223
224 start = time.time()
225 sampling_params = {"max_new_tokens": 1024}
226 response = llm.generate(prompt=text, image_data=image_inputs, sampling_params=sampling_params)
227 print(f"Response costs: {time.time() - start:.2f}s")
228 print(f"Generated text: {response['text']}")
229 ```
230
231 ### Generation Hyperparameters
232 #### VL
233 ```bash
234 export greedy='false'
235 export top_p=0.8
236 export top_k=20
237 export temperature=0.7
238 export repetition_penalty=1.0
239 export presence_penalty=1.5
240 export out_seq_length=16384
241 ```
242
243 #### Text
244 ```bash
245 export greedy='false'
246 export top_p=1.0
247 export top_k=40
248 export repetition_penalty=1.0
249 export presence_penalty=2.0
250 export temperature=1.0
251 export out_seq_length=32768
252 ```
253
254
255
256 ## Citation
257
258 If you find our work helpful, feel free to give us a cite.
259
260 ```
261 @misc{qwen3technicalreport,
262 title={Qwen3 Technical Report},
263 author={Qwen Team},
264 year={2025},
265 eprint={2505.09388},
266 archivePrefix={arXiv},
267 primaryClass={cs.CL},
268 url={https://arxiv.org/abs/2505.09388},
269 }
270
271 @article{Qwen2.5-VL,
272 title={Qwen2.5-VL Technical Report},
273 author={Bai, Shuai and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Song, Sibo and Dang, Kai and Wang, Peng and Wang, Shijie and Tang, Jun and Zhong, Humen and Zhu, Yuanzhi and Yang, Mingkun and Li, Zhaohai and Wan, Jianqiang and Wang, Pengfei and Ding, Wei and Fu, Zheren and Xu, Yiheng and Ye, Jiabo and Zhang, Xi and Xie, Tianbao and Cheng, Zesen and Zhang, Hang and Yang, Zhibo and Xu, Haiyang and Lin, Junyang},
274 journal={arXiv preprint arXiv:2502.13923},
275 year={2025}
276 }
277
278 @article{Qwen2VL,
279 title={Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
280 author={Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Fan, Yang and Dang, Kai and Du, Mengfei and Ren, Xuancheng and Men, Rui and Liu, Dayiheng and Zhou, Chang and Zhou, Jingren and Lin, Junyang},
281 journal={arXiv preprint arXiv:2409.12191},
282 year={2024}
283 }
284
285 @article{Qwen-VL,
286 title={Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond},
287 author={Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren},
288 journal={arXiv preprint arXiv:2308.12966},
289 year={2023}
290 }
291 ```