README.md
6.3 KB · 129 lines · markdown Raw
1 ---
2 license: apache-2.0
3 tags:
4 - vlx
5 - object-detection
6 - grounding
7 - multimodal
8 - REC
9 - VLM
10 - zero-shot-object-detection
11 language:
12 - zh
13 - en
14 pipeline_tag: image-text-to-text
15 library_name: transformers
16 ---
17
18 # VLX-Seek-1.5-10B
19
20 VLX-Seek-1.5-10B is the open-source 10B model in the VLX-Seek 1.5 family, designed for fine-grained perception and visual grounding in embodied scenarios. It targets practical settings such as drones, robots, robotic dogs, surveillance cameras, inspection systems, and other edge-side visual intelligence applications where a model must identify what is present, localize the right instance, and avoid grounding objects that are absent.
21
22 Unlike coordinate-generation-based VLMs that directly decode bounding-box numbers, VLX-Seek reformulates localization as region retrieval and region reference. Candidate visual regions are represented as addressable entities, and the model answers by selecting, comparing, and referring to those regions. This makes localization more aligned with the strengths of language models: comparison, selection, reference, and reasoning.
23
24 For technical details, inference code, and runnable examples, please visit the project repository:
25
26 **GitHub: [om-ai-lab/VLX-Seek](https://github.com/om-ai-lab/VLX-Seek)**
27
28 ## Model Highlights
29
30 - **💡 Embodied visual grounding:** Optimized for real-world embodied scenes, including drone-view, surveillance-view, robot-view, and other edge-side perception scenarios.
31 - **🧩 Region-reference localization:** Converts candidate regions into language-addressable region tokens, reducing reliance on fragile coordinate-string generation.
32 - **💪 Stronger visual capability:** Uses an upgraded visual perception stack with a stronger auxiliary vision tower, improved vision-language alignment, and a stronger VLM backbone.
33 - **🚀 Faster inference design:** Introduces faster OPN proposal generation and more Linear Attention layers to improve inference efficiency and reduce memory usage.
34 - **🔍 Explicit absent-target rejection:** Uses hard-negative rejection training and an explicit `None` output format to reduce hallucinated object grounding.
35 - **📦 Multi-scale family:** VLX-Seek 1.5 is planned in 0.6B, 3B, and 10B sizes. This model card describes the 10B checkpoint.
36
37 ## Intended Use
38
39 VLX-Seek-1.5-10B is intended for research and development in fine-grained multimodal perception, especially:
40
41 - Open-vocabulary object detection and localization.
42 - Referring expression comprehension.
43 - Multi-object visual grounding.
44 - Object counting with region-level evidence.
45 - Drone-view perception with small objects and dense layouts.
46 - Robot-centric and embodied spatial reasoning.
47 - Visual perception modules for edge-side or robot-base-station deployment.
48 - Experiments that require explicit rejection of absent targets.
49
50 The model is especially suitable when the downstream system needs stable region-level anchors rather than only high-level image captions.
51
52 ## How to Use
53
54 VLX-Seek uses a custom region-reference inference pipeline, including candidate-region proposal generation, region-token construction, prompt formatting, and output post-processing. We provide the full inference code and detailed runnable examples in the official repository:
55
56 [https://github.com/om-ai-lab/VLX-Seek](https://github.com/om-ai-lab/VLX-Seek)
57
58 Please follow the repository instructions for installation, model loading, inference scripts, prompt templates, and visualization examples.
59
60
61 ## Evaluation
62
63 ### General Recognition
64
65 <p align="left">
66 <img src="https://om-ai-lab.github.io/images/vlx-seek1.5/Table1_new.jpg" alt="Table 1: Results on common object detection benchmarks." width="70%">
67 </p>
68
69 <p align="left">
70 <img src="https://om-ai-lab.github.io/images/vlx-seek1.5/Table2_new.jpg" alt="Table 2: Results on referring expression comprehension benchmarks." width="70%">
71 </p>
72
73 <p align="left">
74 <img src="https://om-ai-lab.github.io/images/vlx-seek1.5/Table3_new.jpg" alt="Table 3: Results on REC and counting benchmarks." width="70%">
75 </p>
76
77 ### Drone Scenarios
78
79 <p align="left">
80 <img src="https://om-ai-lab.github.io/images/vlx-seek1.5/Table4_new.jpg" alt="Table 4: Results on drone-view benchmarks." width="70%">
81 </p>
82
83 ### Embodied Robot Scenarios
84
85 <p align="left">
86 <img src="https://om-ai-lab.github.io/images/vlx-seek1.5/Table5_new.jpg" alt="Table 5: Results on embodied spatial reasoning benchmarks." width="70%">
87 </p>
88
89 ### Object Hallucination Evaluation
90
91 <p align="left">
92 <img src="https://om-ai-lab.github.io/images/vlx-seek1.5/Table6_new.jpg" alt="Table 6: Object hallucination comparison on HumanRef, VisDrone, and RefDrone." width="70%">
93 </p>
94
95 For full comparison tables and qualitative examples, see the VLX-Seek 1.5 blog:
96
97 [VLX-Seek 1.5: Enhanced Fine-grained Perception for Embodied Scenarios](https://om-ai-lab.github.io/2026_07_06_vlx_seek_1_5_en.html)
98
99 ## Limitations
100
101 - VLX-Seek relies on candidate regions. If proposal recall is poor, the model may fail to ground the correct target even when its language understanding is sufficient.
102 - Region-reference outputs require the VLX-Seek post-processing pipeline to map region tokens back to image coordinates.
103 - The model is optimized for fine-grained visual grounding, but it may still make mistakes under severe occlusion, extreme blur, very low resolution, unusual sensor artifacts, or highly ambiguous referring expressions.
104 - Hard-negative training reduces object hallucination but does not eliminate it. Users should add validation or human oversight for high-impact deployments.
105
106
107 ## Ethical Considerations
108
109 VLX-Seek-1.5-10B can be used in embodied and monitoring scenarios, including cameras, drones, and robots. Such applications may involve privacy, safety, and fairness risks. Users are responsible for complying with applicable laws and institutional policies, obtaining appropriate consent where needed, and avoiding deployments that enable unlawful surveillance, discrimination, or unsafe autonomous actions.
110
111 ## Citation
112
113 If you use VLX-Seek-1.5-10B, please cite the project:
114
115 ```bibtex
116 @misc{vlxseek15_2026,
117 title = {VLX-Seek 1.5: Enhanced Fine-grained Perception for Embodied Scenarios},
118 author = {Om AI Lab},
119 year = {2026},
120 url = {https://om-ai-lab.github.io/2026_07_06_vlx_seek_1_5_en.html}
121 }
122 ```
123
124 ## Contact
125
126 For technical details, inference examples, and updates, please use the official project repository:
127
128 [om-ai-lab/VLX-Seek](https://github.com/om-ai-lab/VLX-Seek)
129