README.md
11.0 KB · 204 lines · markdown Raw
1 ---
2 license: apache-2.0
3 base_model: OpenMed/privacy-filter-nemotron
4 base_model_relation: quantized
5 pipeline_tag: token-classification
6 library_name: gguf
7 datasets:
8 - nvidia/Nemotron-PII
9 tags:
10 - gguf
11 - privacy-filter.cpp
12 - llama-cpp
13 - localai
14 - token-classification
15 - pii
16 - ner
17 - privacy
18 - redaction
19 - nemotron
20 - openai-privacy-filter
21 language:
22 - en
23 ---
24
25 # privacy-filter-nemotron — GGUF (F16 + Q8_0)
26
27 GGUF conversion of [`OpenMed/privacy-filter-nemotron`](https://huggingface.co/OpenMed/privacy-filter-nemotron),
28 a **fine-grained** PII **token-classification** model — a fine-tune of
29 [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) on the
30 [`nvidia/Nemotron-PII`](https://huggingface.co/datasets/nvidia/Nemotron-PII) dataset. It labels
31 every token with a BIOES tag over **55 PII categories (221 classes)** in a single forward pass,
32 then decodes coherent spans with a constrained Viterbi procedure — so it can be served locally
33 with **no Python** as the encoder/NER tier of a PII redactor.
34
35 Where the base [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) covers 8
36 coarse categories, this fine-tune trades multilingual breadth for **category depth**: 55
37 fine-grained English categories (first/last name, government IDs, financial, healthcare,
38 vehicle, digital, …).
39
40 For the full model description, label space, evaluation, limitations, and citations, see the
41 **[source model card](https://huggingface.co/OpenMed/privacy-filter-nemotron)** — this card only
42 covers the GGUF packaging and how to run it.
43
44 > For broader **language** coverage (54 categories across 16 languages) instead of this model's
45 > English-only depth, see the multilingual fine-tune
46 > [`privacy-filter-multilingual` GGUF](https://huggingface.co/LocalAI-io/privacy-filter-multilingual-GGUF).
47
48 ## Runtimes
49
50 This GGUF uses a **custom architecture, `openai-privacy-filter`**, that is not (yet) part of
51 upstream llama.cpp. It runs on:
52
53 1. **[privacy-filter.cpp](https://github.com/localai-org/privacy-filter.cpp)** *(recommended)* —
54 a small standalone GGML engine for exactly this model family, on **stock upstream ggml with
55 no patches** (CPU / CUDA / Vulkan). This is the reference runtime and what the parity numbers
56 below are measured against.
57
58 ```sh
59 # build (see the repo README for CUDA/Vulkan)
60 cmake --preset release && cmake --build --preset release -j
61 # run
62 echo "Contact John Doe at jdoe@example.com" | \
63 build/release/pf-cli --classify privacy-filter-nemotron-f16.gguf 0.5
64 ```
65
66 It exposes a flat C API (`pf_load` / `pf_classify` → entity spans with UTF-8 byte offsets;
67 `pf_tokenize` / `pf_logits`) shaped for FFI — see the repo README.
68
69 2. **[LocalAI](https://github.com/mudler/LocalAI)** — install from the model gallery; LocalAI
70 serves it behind the gRPC `TokenClassify` RPC and runs the constrained BIOES Viterbi decode,
71 returning entity spans. LocalAI drives it through the **`privacy-filter` backend** (which
72 wraps privacy-filter.cpp). The model is **not** a chat/completion model — it is a PII detector
73 that other models opt into via a `pii.detectors` list.
74
75 3. **llama.cpp — only with a patch.** Stock `llama.cpp`, `llama-cpp-python`, Ollama, and
76 LM Studio will **fail to load** this file (`unknown model architecture:
77 'openai-privacy-filter'`). The arch can be added with carry-patches (TOKEN_CLS pooling, the
78 architecture + HF→GGUF converter, the bidirectional banded-attention graph, and an all-SWA
79 no-cache mask fix; TOKEN_CLS pooling tracks the still-open
80 [PR #19725](https://github.com/ggml-org/llama.cpp/pull/19725)). Until that support lands
81 upstream, `privacy-filter.cpp` above is the patch-free alternative.
82
83 > **Pooling note (llama.cpp path only):** the model must be loaded with **TOKEN_CLS pooling**
84 > (the GGUF's default). If you drive `llama-embedding` directly for testing, do **not** pass
85 > `--pooling none` — that overrides the default and yields raw hidden states instead of label
86 > logits. privacy-filter.cpp handles this automatically.
87
88 ## Files
89
90 | File | Precision | Size | Notes |
91 |---|---|---|---|
92 | `privacy-filter-nemotron-f16.gguf` | F16 | 2.82 GB | Reference artifact. 156 tensors; 221 `classifier.output_labels`; `pooling_type = TOKEN_CLS`. |
93 | `privacy-filter-nemotron-q8.gguf` | Q8_0 (experts) | 1.64 GB | MoE expert weights → Q8_0, the rest F16. For RAM-constrained / edge use. |
94
95 ```
96 sha256 (f16): 70dfe91ff220ff04594168a83e296dcc2054449cde77f98d0e782edbb6a31f5a
97 sha256 (q8): 2ec11c154e572a2686f4d77e861b7f74e6917e09638fe9bd27156d48bd99e21a
98 ```
99
100 **Q8_0 quantization — and why it isn't free.** `q8` stores the bulk of the weights (the MoE
101 expert matrices) as 8-bit integers instead of 16-bit floats — via
102 [`scripts/requant_q8.py`](https://github.com/localai-org/privacy-filter.cpp/blob/master/scripts/requant_q8.py),
103 with attention, embeddings and the classifier head left at F16. That cuts the download by ~42%
104 (2.82 GB → 1.64 GB) and is usually a bit faster on CPU.
105
106 The catch: **reducing precision throws information away, and it is almost never a free lunch.**
107 On a mixed-PII document (1,360 tokens) q8 matched f16 on **99.93%** of token labels (1,359/1,360)
108 and produced an **identical span set** at threshold 0.5, with an average prediction shift (KL
109 divergence) of just **2.6e-5** — but note it did **not** match on all tokens; one token flipped.
110 That is the point in miniature: a reassuring average still hides the specific cases that change,
111 and **accuracy benchmarks routinely look fine right up until the one that bites.** Those numbers
112 also come from a single English document; a tiny *average* shift can still hide a flip on the one
113 input that matters to you — a rare name, an unusual ID format, or one of the fuzzier categories
114 below. For PII detection a single missed span is a leak, so:
115
116 - **Prefer F16** if you can afford the 2.82 GB — it is the reference these numbers are measured
117 against, and what we trust by default.
118 - **Use Q8_0** when memory or speed forces it (e.g. a 4 GB Raspberry Pi 5), treat it as a
119 deliberate size/speed tradeoff, and **validate it on your own data** first.
120
121 ## Architecture & conversion
122
123 gpt-oss-style sparse **MoE** (8 layers, `d_model=640`, 128 experts, top-4 routing; ~1.5B total /
124 ~50M active per token), **bidirectional banded attention** (symmetric sliding window 128,
125 attention sinks retained), **interleaved (GPT-J) RoPE** with YaRN (θ=150000, factor 32), o200k
126 (`o200k_base`) tokenizer, and a 221-way token-classification head (`score` → `cls.output`). The
127 architecture is identical to the rest of the `privacy-filter` family — only the fine-tuned
128 weights and the larger (221-class) head differ.
129
130 The conversion reproduces the unmodified `transformers` reference at F16: across the parity
131 prompt set (short / PII-dense / multilingual / a 3k-token document) the F16 GGUF agrees with HF
132 on **99.94%** of per-token argmaxes — **100% up to ~300 tokens**, with the only two flips being
133 argmax *ties* at ~3k positions (the F16-rounding regime) — at **full-logit cosine ≥ 0.9995
134 (mean 0.999997)**. A wrong expert transpose would crater that cosine, so the two load-bearing
135 conversion choices — the expert `gate_up` `chunk(2)` split and the `n_swa = 2·sliding_window`
136 window mapping — are confirmed by it. privacy-filter.cpp re-derives the YaRN `truncate=false`
137 frequencies at load time (fed to `ggml_rope_ext` as `freq_factors`) so the same GGUF is
138 interchangeable across runtimes.
139
140 This GGUF was produced by [`scripts/convert.py`](https://github.com/localai-org/privacy-filter.cpp/blob/master/scripts/convert.py)
141 — a self-contained HF→GGUF converter (no llama.cpp dependency). The same converter is re-run by
142 CI and gated against the HF reference logits for the sibling models, so the published artifact
143 stays in parity.
144
145 ## Label space
146
147 `O` plus `B-`/`I-`/`E-`/`S-` for each of 55 categories (1 + 55×4 = 221), spanning identity,
148 contact, address, dates/time, government IDs, financial, healthcare, enterprise IDs, vehicle, and
149 digital entities:
150
151 - **Identity** — `first_name`, `last_name`, `user_name`, `age`, `gender`, `race_ethnicity`,
152 `sexuality`, `religious_belief`, `political_view`, `education_level`, `occupation`,
153 `employment_status`, `language`, `blood_type`, `biometric_identifier`
154 - **Contact** — `email`, `phone_number`, `fax_number`, `url`
155 - **Address** — `street_address`, `city`, `county`, `state`, `country`, `postcode`, `coordinate`
156 - **Dates** — `date`, `date_of_birth`, `date_time`, `time`
157 - **Government IDs** — `ssn`, `national_id`, `tax_id`
158 - **Financial** — `account_number`, `bank_routing_number`, `swift_bic`, `credit_debit_card`,
159 `cvv`, `pin`, `password`
160 - **Healthcare** — `medical_record_number`, `health_plan_beneficiary_number`
161 - **Enterprise** — `company_name`, `customer_id`, `employee_id`, `unique_id`,
162 `certificate_license_number`
163 - **Vehicle** — `license_plate`, `vehicle_identifier`
164 - **Digital** — `ipv4`, `ipv6`, `mac_address`, `device_identifier`, `api_key`, `http_cookie`
165
166 The ordered `id2label` table is embedded in the GGUF (`classifier.output_labels`). See the
167 [source card](https://huggingface.co/OpenMed/privacy-filter-nemotron#label-space-55-categories)
168 for the canonical grouping.
169
170 ## Evaluation
171
172 From the [source model card](https://huggingface.co/OpenMed/privacy-filter-nemotron#performance)
173 (on the Nemotron-PII test split): **macro B-F1 = 0.9533**, **token accuracy = 0.9910**, with
174 **46/55** labels at F1 ≥ 0.90, **7/55** in 0.70–0.89, and **none** below 0.70. The fuzzier,
175 more subjective categories (`occupation`, `language`, `gender`, `state`, `race_ethnicity`,
176 `political_view`, `education_level`) sit lowest (F1 ≈ 0.65–0.89) vs the strictly-formatted
177 identifiers (≥ 0.95). The GGUF reproduces the HF logits at F16, so these numbers carry over.
178
179 ## Limitations & intended use
180
181 Identical to the [source model](https://huggingface.co/OpenMed/privacy-filter-nemotron#limitations--intended-use):
182
183 - **English-only.** Nemotron-PII is predominantly English (50/50 US/international locale split);
184 non-English performance is not guaranteed. For multilingual text, prefer the
185 [multilingual fine-tune](https://huggingface.co/LocalAI-io/privacy-filter-multilingual-GGUF).
186 - **Synthetic training data.** Nemotron-PII is synthesized; real clinical notes, legal documents,
187 and web text may show different surface forms — collect a domain eval set and re-calibrate
188 thresholds for high-stakes use.
189 - **Fuzzier categories are weaker.** Treat low-confidence predictions on the subjective
190 categories above accordingly.
191 - **Not a substitute for legal/compliance review**, and **not** a clinical PHI model. Use it as
192 one tier behind deterministic regex pre-filters and human review.
193
194 ## License
195
196 **Apache-2.0**, inherited from `openai/privacy-filter` and `OpenMed/privacy-filter-nemotron`.
197
198 ## Credits & citation
199
200 Conversion and runtime support by the **LocalAI** project (`privacy-filter.cpp`). The model
201 itself is by **OpenMed**, fine-tuned from **OpenAI**'s `privacy-filter` on **NVIDIA**'s
202 **Nemotron-PII** dataset — please cite all of them (BibTeX in the
203 [source card](https://huggingface.co/OpenMed/privacy-filter-nemotron#citation)).
204