README.md
3.8 KB · 100 lines · markdown Raw
1 ---
2 license: other
3 license_name: nvidia-license
4 license_link: https://huggingface.co/nvidia/LocateAnything-3B
5 base_model: nvidia/LocateAnything-3B
6 tags:
7 - locate-anything.cpp
8 - ggml
9 - gguf
10 - object-detection
11 - open-vocabulary-detection
12 - visual-grounding
13 - localai
14 pipeline_tag: object-detection
15 library_name: gguf
16 ---
17
18 # locate-anything.cpp - GGUF
19
20 GGUF builds of [`nvidia/LocateAnything-3B`](https://huggingface.co/nvidia/LocateAnything-3B)
21 for **[locate-anything.cpp](https://github.com/mudler/locate-anything.cpp)** - a C++/ggml
22 inference engine for open-vocabulary detection / visual grounding, no Python at inference time.
23
24 **Brought to you by the [LocalAI](https://github.com/mudler/LocalAI) team.**
25
26 The detections are the same as the official PyTorch implementation (the engine is
27 parity-gated against it), and it runs faster - on CPU and GPU.
28
29 ## Files
30
31 | File | Bits (LM) | Size | Notes |
32 | ---- | --------- | ---- | ----- |
33 | `locate-anything-f16.gguf` | f16 | ~9.2 GB | LM matmuls in f16, everything else f32 |
34 | `locate-anything-q8_0.gguf` | q8_0 | ~6.3 GB | near-lossless; **box-identical** to f32 - recommended |
35 | `locate-anything-q6_k.gguf` | q6_k | ~5.5 GB | box-identical to f32 |
36 | `locate-anything-q5_k.gguf` | q5_k | ~5.1 GB | sub-pixel box drift |
37 | `locate-anything-q4_k.gguf` | q4_k | ~4.7 GB | smallest; sub-pixel box drift |
38
39 The full-precision `f32` GGUF (~15 GB) is reproducible from the HF weights with
40 `scripts/convert_locateanything_to_gguf.py` in the repo.
41
42 ## Performance
43
44 Same detections as the official model, faster. Full methodology, the warm/median setup,
45 parity checks, and more images are in the repo's
46 [`benchmarks/BENCHMARK.md`](https://github.com/mudler/locate-anything.cpp/blob/master/benchmarks/BENCHMARK.md).
47
48 ### Quantization (CPU, Ryzen 9 9950X3D)
49
50 Slow-mode inference on the 448 fixture; `vs official` divides the official PyTorch **f32**
51 time (23.65 s) by each. Only the Qwen2 LM matmuls are quantized, so box parity is preserved
52 through q6_k:
53
54 | dtype | size | infer | vs official f32 | boxes |
55 | ----- | ---- | ----- | --------------- | ----- |
56 | f16 | 9.15 GB | 13.68 s | 1.7× | identical |
57 | q8_0 | 6.26 GB | 6.07 s | **3.9×** | identical |
58 | q6_k | 5.51 GB | 5.77 s | **4.1×** | identical |
59 | q5_k | 5.10 GB | 5.11 s | **4.6×** | sub-pixel |
60 | q4_k | 4.72 GB | 4.29 s | **5.5×** | sub-pixel |
61
62 ![quantization size vs speedup](quant_tradeoff.png)
63
64 ### GPU (NVIDIA GB10, vs the official bf16 model)
65
66 Run against the official model exactly as its model card documents (bf16), greedily, on one
67 GB10 GPU. Precision-matched (our **f16** vs its bf16) ours is **~1.7×** faster; the
68 recommended **q8_0** build (box-identical) is **~1.9-2.1×**:
69
70 ![GB10 GPU speedup vs official bf16](gpu_speedup.png)
71
72 ## Quantization policy
73
74 Only the Qwen2 language-model matmuls (`attn_{q,k,v,o}`, `ffn_{gate,up,down}`, `lm.output`)
75 are quantized. The MoonViT vision tower, the projector, all norms and biases, and the two
76 host-read f32 tensors (`lm.tok_embd`, `vit.pos_emb`) stay **f32** - so the parity-sensitive
77 vision path is untouched. q8_0/q6_k are box-identical; lower bit-widths trade a little box
78 precision for size.
79
80 ## Usage
81
82 ```sh
83 # build the CLI (see the repo README), then:
84 locate-anything-cli detect \
85 --model locate-anything-q8_0.gguf \
86 --input image.jpg \
87 --prompt "Locate all the instances that matches the following description: person</c>car." \
88 --annotated out.png
89 # -> {"detections":[{"label":"person","box":[...]}, ...]} + an annotated PNG
90 ```
91
92 Decode modes: `--mode hybrid` (default), `slow`, `fast`. GPU: build with `-DLA_GGML_CUDA=ON`
93 and run with `LA_DEVICE=` (auto-GPU). Separate categories in the prompt with `</c>`.
94
95 ## License
96
97 The model weights are NVIDIA's, distributed under
98 [NVIDIA's license](https://huggingface.co/nvidia/LocateAnything-3B); this repository
99 redistributes them in GGUF form for use with locate-anything.cpp (MIT).
100