README.md
10.8 KB · 335 lines · markdown Raw
1 ---
2 pipeline_tag: image-text-to-text
3 language:
4 - multilingual
5 tags:
6 - baidu
7 - vision-language
8 - ocr
9 - custom_code
10 license: mit
11 library_name: transformers
12 ---
13 <p align="center">
14 <img src="assets/baidu.png" width="55%" alt="Baidu Inc." />
15 </p>
16
17 <hr>
18
19 <h1 align="center">Unlimited OCR Works</h1>
20
21 <div align="center">
22
23 <a href="https://trendshift.io/repositories/62053?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-62053" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/62053/daily" alt="baidu%2FUnlimited-OCR | Trendshift" width="250" height="55"/></a>
24
25 <a href="https://github.com/baidu/Unlimited-OCR">
26 <img alt="GitHub" src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white" />
27 </a>
28 <a href="https://huggingface.co/baidu/Unlimited-OCR">
29 <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-ffc107?color=ffc107&logoColor=white" />
30 </a>
31 </div>
32
33 <div align="center">
34 <a href="https://arxiv.org/abs/2606.23050">
35 <img alt="arXiv" src="https://img.shields.io/badge/arXiv-Unlimited OCR Works-b31b1b?logo=arxiv&logoColor=white" />
36 </a>
37 <a href="https://x.com/Baidu_Inc" target="_blank">
38 <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-Baidu Inc.-white?logo=x&logoColor=white" />
39 </a>
40 </div>
41
42 <h3 align="center">Welcome the Era of One-shot Long-horizon Parsing.</h3>
43
44 <p align="center">
45 <img src="assets/Unlimited-OCR.png" width="1000" alt="Unlimited OCR overview" />
46 </p>
47
48
49 ## Release
50 - [2026/07/21] 🤝 Thanks to the [ms-swift community](https://github.com/modelscope/ms-swift) for their support, our model now supports training with [ms-swift](https://github.com/modelscope/ms-swift).
51 - [2026/07/03] 🤝 Thanks to the Baidu Cloud team for their support. Our model is now available on [Baidu Cloud](https://cloud.baidu.com/doc/OCR/s/fmr1p39gb).
52 - [2026/06/28] 🤝 Thanks to the [vLLM community](https://github.com/vllm-project/vllm) and [Tianyu Guo](https://github.com/gty111) for their support, our model now supports vLLM inference.
53 - [2026/06/24] 🤝 Thanks to [AK](https://x.com/_akhaliq) for creating a demo for us. It is now available at [Hugging Face Spaces](https://huggingface.co/spaces/baidu/Unlimited-OCR).
54 - [2026/06/23] 📄 Our paper is now available on [arXiv](https://arxiv.org/abs/2606.23050).
55 - [2026/06/23] 🤝 Thanks to the [ModelScope community](https://github.com/modelscope) for their support. Our model is now available at [ModelScope](https://modelscope.cn/models/PaddlePaddle/Unlimited-OCR).
56 - [2026/06/22] 🚀 We present [Unlimited-OCR](https://github.com/baidu/Unlimited-OCR), aiming to push [Deepseek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) one step further.
57
58 ## Inference
59
60 ### Transformers
61 Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.12.3 + CUDA12.9：
62
63 ```
64 torch==2.10.0
65 torchvision==0.25.0
66 transformers==4.57.1
67 Pillow==12.1.1
68 matplotlib==3.10.8
69 einops==0.8.2
70 addict==2.4.0
71 easydict==1.13
72 pymupdf==1.27.2.2
73 psutil==7.2.2
74 ```
75
76 ```python
77 import os
78 import torch
79 from transformers import AutoModel, AutoTokenizer
80
81 model_name = 'baidu/Unlimited-OCR'
82
83 tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
84 model = AutoModel.from_pretrained(
85 model_name,
86 trust_remote_code=True,
87 use_safetensors=True,
88 torch_dtype=torch.bfloat16,
89 )
90 model = model.eval().cuda()
91
92 # ── Single image supports two configs: gundam or base ──
93 # gundam: base_size=1024, image_size=640, crop_mode=True
94 # base: base_size=1024, image_size=1024, crop_mode=False
95 model.infer(
96 tokenizer,
97 prompt='<image>document parsing.',
98 image_file='your_image.jpg',
99 output_path='your/output/dir',
100 base_size=1024, image_size=640, crop_mode=True,
101 max_length=32768,
102 no_repeat_ngram_size=35, ngram_window=128,
103 save_results=True,
104 )
105
106 # ── Multi page / PDF only uses base (image_size=1024) ──
107 model.infer_multi(
108 tokenizer,
109 prompt='<image>Multi page parsing.',
110 image_files=['page1.png', 'page2.png', 'page3.png'],
111 output_path='your/output/dir',
112 image_size=1024,
113 max_length=32768,
114 no_repeat_ngram_size=35, ngram_window=1024,
115 save_results=True,
116 )
117
118 # ── PDF (convert pages to images, then multi-page parsing) ──
119 import tempfile, fitz # PyMuPDF
120
121 def pdf_to_images(pdf_path, dpi=300):
122 doc = fitz.open(pdf_path)
123 tmp_dir = tempfile.mkdtemp(prefix='pdf_ocr_')
124 mat = fitz.Matrix(dpi / 72, dpi / 72)
125 paths = []
126 for i, page in enumerate(doc):
127 out = os.path.join(tmp_dir, f'page_{i+1:04d}.png')
128 page.get_pixmap(matrix=mat).save(out)
129 paths.append(out)
130 doc.close()
131 return paths
132
133 model.infer_multi(
134 tokenizer,
135 prompt='<image>Multi page parsing.',
136 image_files=pdf_to_images('your_doc.pdf', dpi=300),
137 output_path='your/output/dir',
138 image_size=1024,
139 max_length=32768,
140 no_repeat_ngram_size=35, ngram_window=1024,
141 save_results=True,
142 )
143 ```
144
145 ### vLLM
146
147 Please refer to the official vLLM recipe for deployment details:
148
149 **Recipe:** [https://recipes.vllm.ai/baidu/Unlimited-OCR](https://recipes.vllm.ai/baidu/Unlimited-OCR)
150
151 ##### Docker Images
152 Use the following Docker images depending on your GPU platform:
153
154 **Default (CUDA 13.0):**
155 ```bash
156 docker pull vllm/vllm-openai:unlimited-ocr
157 ```
158 **For Hopper GPUs (CUDA 12.9)**
159 ```bash
160 docker pull vllm/vllm-openai:unlimited-ocr-cu129
161 ```
162
163 ### SGLang
164
165 Set up the environment (uv-managed virtualenv). Install the local SGLang wheel first,
166 then pin `kernels==0.9.0` and install PyMuPDF for PDF-to-image conversion:
167 ```shell
168 uv venv --python 3.12
169 source .venv/bin/activate
170
171 uv pip install wheel/sglang-0.0.0.dev11416+g92e8bb79e-py3-none-any.whl
172 uv pip install kernels==0.11.7
173 uv pip install pymupdf==1.27.2.2
174 ```
175
176 Start the SGLang server:
177 ```shell
178 python -m sglang.launch_server \
179 --model baidu/Unlimited-OCR \
180 --served-model-name Unlimited-OCR \
181 --attention-backend fa3 \
182 --page-size 1 \
183 --mem-fraction-static 0.8 \
184 --context-length 32768 \
185 --enable-custom-logit-processor \
186 --disable-overlap-schedule \
187 --skip-server-warmup \
188 --host 0.0.0.0 \
189 --port 10000
190 ```
191
192 Send streaming requests to the OpenAI-compatible API:
193 ```python
194 import base64
195 import json
196 import os
197 import tempfile
198
199 import fitz
200 import requests
201 from sglang.srt.sampling.custom_logit_processor import DeepseekOCRNoRepeatNGramLogitProcessor
202
203 server_url = "http://127.0.0.1:10000"
204
205 session = requests.Session()
206 session.trust_env = False
207
208
209 def pdf_to_images(pdf_path, dpi=300):
210 doc = fitz.open(pdf_path)
211 tmp_dir = tempfile.mkdtemp(prefix="pdf_ocr_")
212 mat = fitz.Matrix(dpi / 72, dpi / 72)
213 image_paths = []
214 for i, page in enumerate(doc):
215 image_path = os.path.join(tmp_dir, f"page_{i + 1:04d}.png")
216 page.get_pixmap(matrix=mat).save(image_path)
217 image_paths.append(image_path)
218 doc.close()
219 return image_paths
220
221
222 def encode_image(image_path):
223 ext = os.path.splitext(image_path)[1].lower()
224 mime = "image/jpeg" if ext in (".jpg", ".jpeg") else f"image/{ext.lstrip('.')}"
225 with open(image_path, "rb") as f:
226 data = base64.b64encode(f.read()).decode("utf-8")
227 return {"type": "image_url", "image_url": {"url": f"data:{mime};base64,{data}"}}
228
229
230 def build_content(prompt, image_paths):
231 return [{"type": "text", "text": prompt}] + [encode_image(path) for path in image_paths]
232
233
234 def generate(prompt, image_paths, image_mode, ngram_window):
235 payload = {
236 "model": "Unlimited-OCR",
237 "messages": [{"role": "user", "content": build_content(prompt, image_paths)}],
238 "temperature": 0,
239 "skip_special_tokens": False,
240 "images_config": {"image_mode": image_mode},
241 "custom_logit_processor": DeepseekOCRNoRepeatNGramLogitProcessor.to_str(),
242 "custom_params": {
243 "ngram_size": 35,
244 "window_size": ngram_window,
245 },
246 "stream": True,
247 }
248 response = session.post(
249 f"{server_url}/v1/chat/completions",
250 headers={"Content-Type": "application/json"},
251 data=json.dumps(payload),
252 timeout=1200,
253 stream=True,
254 )
255 response.raise_for_status()
256
257 chunks = []
258 for line in response.iter_lines(chunk_size=1, decode_unicode=True):
259 if not line or not line.startswith("data: "):
260 continue
261 data = line[len("data: "):]
262 if data == "[DONE]":
263 break
264 event = json.loads(data)
265 delta = event["choices"][0].get("delta", {}).get("content", "")
266 if delta:
267 print(delta, end="", flush=True)
268 chunks.append(delta)
269 print()
270 return "".join(chunks)
271
272
273 # Single image supports two configs: gundam or base. Example below uses gundam.
274 generate("document parsing.", ["your_image.jpg"], image_mode="gundam", ngram_window=128)
275
276 # Multi image (base only)
277 generate("Multi page parsing.", ["page1.png", "page2.png"], image_mode="base", ngram_window=1024)
278
279 # PDF (base only)
280 generate("Multi page parsing.", pdf_to_images("your_doc.pdf", dpi=300), image_mode="base", ngram_window=1024)
281 ```
282
283 For OmniDocBench evaluation, you need to perform the following post-processing.
284 ```python
285 DET_RE = re.compile(r'<\|det\|>([^<\s]+)(?:\s*\[[^\]]*\])?\s*<\|/det\|>(.*)', re.DOTALL)
286
287 def remove_det(raw: str) -> str:
288 """
289 Strip <|det|>type [bbox]<|/det|> markers, group lines belonging to the
290 same block with \\n, and separate different blocks with \\n\\n.
291 """
292 blocks = []
293 cur = None
294 for line in raw.splitlines():
295 line = line.rstrip()
296 if not line:
297 continue
298 m = DET_RE.match(line)
299 if m:
300 category, content = m.group(1).strip(), m.group(2).strip()
301 if category == 'image':
302 continue
303 if cur is not None:
304 blocks.append(cur)
305 cur = [content] if content else []
306 continue
307 if cur is None:
308 cur = []
309 cur.append(line)
310 if cur is not None:
311 blocks.append(cur)
312 text = '\n\n'.join('\n'.join(b) for b in blocks).strip()
313 return text
314 ```
315
316
317 ## Visualization
318
319 <img src="assets/long-horizon-ocr.gif" width="100%" alt="Long-horizon OCR demo" />
320
321 ## Acknowledgement
322
323 We would like to thank [Deepseek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR), [Deepseek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2), [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) for their valuable models and ideas.
324
325 ## Citation
326 ```bibtex
327 @misc{yin2026unlimitedocrworks,
328 title={Unlimited OCR Works},
329 author={Youyang Yin and Huanhuan Liu and YY and Qunyi Xie and Chaorun Liu and Shiqi Yang and Shaohua Wang and Zhanlong Liu and Hao Zou and Jinyue Chen and Shu Wei and Jingjing Wu and Mingxin Huang and Zhen Wu and Guibin Wang and Tengyu Du and Lei Jia},
330 year={2026},
331 eprint={2606.23050},
332 archivePrefix={arXiv},
333 primaryClass={cs.CV},
334 url={https://arxiv.org/abs/2606.23050},
335 }