README.md
8.5 KB · 183 lines · markdown Raw
1 ---
2 license: apache-2.0
3 base_model: OpenMed/privacy-filter-multilingual
4 base_model_relation: quantized
5 pipeline_tag: token-classification
6 library_name: gguf
7 tags:
8 - gguf
9 - privacy-filter.cpp
10 - llama-cpp
11 - localai
12 - token-classification
13 - pii
14 - ner
15 - privacy
16 - redaction
17 - multilingual
18 - openai-privacy-filter
19 language:
20 - ar
21 - bn
22 - de
23 - en
24 - es
25 - fr
26 - hi
27 - it
28 - ja
29 - ko
30 - nl
31 - pt
32 - te
33 - tr
34 - vi
35 - zh
36 ---
37
38 # privacy-filter-multilingual — GGUF (F16 + Q8_0)
39
40 GGUF conversion of [`OpenMed/privacy-filter-multilingual`](https://huggingface.co/OpenMed/privacy-filter-multilingual),
41 a multilingual PII **token-classification** model (a fine-tune of
42 [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)). It labels every
43 token with a BIOES tag over **54 PII categories (217 classes)** across **16 languages**, so
44 it can be served locally with **no Python** as the encoder/NER tier of a PII redactor.
45
46 For the full model description, label space, evaluation, limitations, and citations, see the
47 **[source model card](https://huggingface.co/OpenMed/privacy-filter-multilingual)** — this
48 card only covers the GGUF packaging and how to run it.
49
50 ## Runtimes
51
52 This GGUF uses a **custom architecture, `openai-privacy-filter`**, that is not (yet) part of
53 upstream llama.cpp. It runs on:
54
55 1. **[privacy-filter.cpp](https://github.com/localai-org/privacy-filter.cpp)** *(recommended)* —
56 a small standalone GGML engine for exactly this model family, on **stock upstream ggml with
57 no patches** (CPU / CUDA / Vulkan). This is the reference runtime and what the parity numbers
58 below are measured against.
59
60 ```sh
61 # build (see the repo README for CUDA/Vulkan)
62 cmake --preset release && cmake --build --preset release -j
63 # run
64 echo "Contact John Doe at jdoe@example.com" | \
65 build/release/pf-cli --classify privacy-filter-multilingual-f16.gguf 0.5
66 ```
67
68 It exposes a flat C API (`pf_load` / `pf_classify` → entity spans with UTF-8 byte offsets;
69 `pf_tokenize` / `pf_logits`) shaped for FFI — see the repo README.
70
71 2. **[LocalAI](https://github.com/mudler/LocalAI)** — install from the model gallery; LocalAI
72 serves it behind the gRPC `TokenClassify` RPC and runs the constrained BIOES Viterbi decode,
73 returning entity spans. LocalAI drives it through the **`privacy-filter` backend** (which
74 wraps privacy-filter.cpp); older builds used a llama.cpp-patched path. The model is **not** a
75 chat/completion model — it is a PII detector that other models opt into.
76
77 ```bash
78 local-ai models install privacy-filter-multilingual
79 ```
80
81 The gallery entry carries the detection policy in a `pii_detection:` block (default: mask
82 everything detected; block credentials / financial-secrets / crypto). Other models opt in by
83 listing it under `pii.detectors`:
84
85 ```yaml
86 # any chat or cloud-proxy model — opt in and reference the detector(s)
87 name: my-assistant
88 pii:
89 enabled: true
90 detectors:
91 - privacy-filter-multilingual
92 ```
93
94 3. **llama.cpp — only with a patch.** Stock `llama.cpp`, `llama-cpp-python`, Ollama, and
95 LM Studio will **fail to load** this file (`unknown model architecture:
96 'openai-privacy-filter'`). The arch can be added with carry-patches (TOKEN_CLS pooling, the
97 architecture + HF→GGUF converter, the bidirectional banded-attention graph, and an all-SWA
98 no-cache mask fix; TOKEN_CLS pooling tracks the still-open
99 [PR #19725](https://github.com/ggml-org/llama.cpp/pull/19725)). Until that support lands
100 upstream, the patched path is carried by LocalAI; `privacy-filter.cpp` above is the
101 patch-free alternative.
102
103 > **Pooling note (llama.cpp path only):** the model must be loaded with **TOKEN_CLS pooling**
104 > (the GGUF's default). If you drive `llama-embedding` directly for testing, do **not** pass
105 > `--pooling none` — that overrides the default and yields raw hidden states instead of label
106 > logits. privacy-filter.cpp handles this automatically.
107
108 ## Files
109
110 | File | Precision | Size | Notes |
111 |---|---|---|---|
112 | `privacy-filter-multilingual-f16.gguf` | F16 | ~2.7 GB | Reference artifact. 217 `classifier.output_labels`; `pooling_type = TOKEN_CLS`. |
113 | `privacy-filter-multilingual-q8.gguf` | Q8_0 (experts) | ~1.6 GB | MoE expert weights → Q8_0, the rest F16. For RAM-constrained / edge use. |
114
115 `sha256 (q8): 968135172ba8202374b4c3bd7d353e100c8fc574035da793fa4d13ca441319b7`
116
117 **Q8_0 quantization — and why it isn't free.** `q8` stores the bulk of the weights (the MoE
118 expert matrices) as 8-bit integers instead of 16-bit floats — via
119 [`scripts/requant_q8.py`](https://github.com/localai-org/privacy-filter.cpp/blob/master/scripts/requant_q8.py),
120 with attention, embeddings and the classifier head left at F16. That roughly halves the download
121 (≈2.7 GB → ≈1.6 GB) and is usually a bit faster on CPU.
122
123 The catch: **reducing precision throws information away, and it is almost never a free lunch.**
124 Our checks didn't find a regression — on a mixed-PII document (1,360 tokens) q8 matched f16 on
125 **100%** of token labels and produced identical spans, with an average prediction shift
126 (KL divergence) of just 6.9e-5. But "we didn't find a difference" is not the same as "there is
127 none." Those numbers come from a single English document, and a tiny *average* shift can still
128 hide a flip on the one input that matters to you — a rare name, an unusual phone or ID format, or
129 a language we never tested. **Accuracy benchmarks and divergence metrics routinely look
130 reassuring right up until the case that bites.** For PII detection a single missed span is a
131 leak, so:
132
133 - **Prefer F16** if you can afford the ~2.7 GB — it is the reference these numbers are measured
134 against, and what we trust by default.
135 - **Use Q8_0** when memory or speed forces it (e.g. a 4 GB Raspberry Pi 5), treat it as a
136 deliberate tradeoff, and **validate it on your own data** first. A full span-F1-per-language
137 sweep across the 16 languages is the bar we'd want before calling q8 a true drop-in.
138
139 ## Architecture & conversion
140
141 gpt-oss-style sparse **MoE** (8 layers, `d_model=640`, 128 experts, top-4 routing, ~50M active
142 per token), **bidirectional banded attention** (symmetric sliding window 128, attention sinks
143 retained), **interleaved (GPT-J) RoPE** with YaRN (θ=150000, factor 32), o200k (`o200k_base`)
144 tokenizer, and a 217-way token-classification head (`score` → `cls.output`).
145
146 The conversion reproduces the HF reference **exactly at F16**: token-for-token argmax match on
147 the parity prompt set, **full-logit cosine = 1.0**, every layer's residual-stream cosine = 1.0
148 (relerr ≈ 2e-4, i.e. F16 rounding). The two load-bearing conversion choices — the expert
149 `gate_up` `chunk(2)` split and the `n_swa = 2·sliding_window` window mapping — are both
150 confirmed by that parity. privacy-filter.cpp re-derives the YaRN `truncate=false` frequencies at
151 load time (fed to `ggml_rope_ext` as `freq_factors`) so the same GGUF is interchangeable across
152 runtimes.
153
154 This GGUF was produced by [`scripts/convert.py`](https://github.com/localai-org/privacy-filter.cpp/blob/master/scripts/convert.py)
155 — a self-contained HF→GGUF converter (no llama.cpp dependency). Nightly CI re-runs it and gates
156 the output against the HF reference logits, so the published artifact stays in parity.
157
158 ## Label space
159
160 `O` plus `B-`/`I-`/`E-`/`S-` for each of 54 categories (1 + 54×4 = 217), spanning identity,
161 contact, address, dates/time, government IDs, financial, crypto, vehicle, digital, and auth
162 entities. The ordered `id2label` table is embedded in the GGUF (`classifier.output_labels`).
163 See the [source card](https://huggingface.co/OpenMed/privacy-filter-multilingual#label-space-54-categories)
164 for the full list.
165
166 ## Limitations & intended use
167
168 Identical to the [source model](https://huggingface.co/OpenMed/privacy-filter-multilingual#limitations--intended-use):
169 multilingual but uneven (strongest on de/es/fr/it/hi/te/en; weaker on CJK), trained on
170 synthetic AI4Privacy data, **not** a substitute for legal/compliance review, and **not** a
171 clinical PHI model. Use it as one tier behind deterministic regex pre-filters and human review.
172
173 ## License
174
175 **Apache-2.0**, inherited from `openai/privacy-filter` and `OpenMed/privacy-filter-multilingual`.
176
177 ## Credits & citation
178
179 Conversion and runtime support by the **LocalAI** project (`privacy-filter.cpp`). The model
180 itself is by **OpenMed**, fine-tuned from **OpenAI**'s `privacy-filter`, on **AI4Privacy**
181 datasets — please cite all of them (BibTeX in the
182 [source card](https://huggingface.co/OpenMed/privacy-filter-multilingual#citation)).
183