README.md · bert-small-pii-detection

1

---

2

license: apache-2.0

3

datasets:

4

- gravitee-io/pii-detection-dataset

5

language:

6

- en

7

base_model:

8

- prajjwal1/bert-small

9

pipeline_tag: token-classification

10

tags:

11

- pii

12

- ner

13

- token-classification

14

- privacy

15

- ai-gateway

16

---

17

# gravitee-io/bert-small-pii-detection 🚀

18

19

Token-classification model for PII detection, fine-tuned from `prajjwal1/bert-small` on

20

[`gravitee-io/pii-detection-dataset`](https://huggingface.co/datasets/gravitee-io/pii-detection-dataset).

21

22

### Label Set

23

24

```

25

AGE, COORDINATE, CREDIT_CARD, DATE_TIME, EMAIL_ADDRESS, FINANCIAL, HONORIFIC, IBAN_CODE, IMEI,

26

IP_ADDRESS, LOCATION, MAC_ADDRESS, NRP, ORGANIZATION, PASSWORD, PERSON, PHONE_NUMBER,

27

TITLE, URL, US_BANK_NUMBER, US_DRIVER_LICENSE, US_ITIN, US_LICENSE_PLATE, US_PASSPORT, US_SSN

28

```

29

30

## How to Use

31

32

### Quick start (pipeline)

33

34

```python

35

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

36

37

repo = "gravitee-io/bert-small-pii-detection"

38

tok = AutoTokenizer.from_pretrained(repo)

39

model = AutoModelForTokenClassification.from_pretrained(repo)

40

41

pipe = pipeline("token-classification", model=model, tokenizer=tok, aggregation_strategy="simple")

42

text = "Contact John Smith at john@example.com"

43

pipe(text)

44

```

45

46

### ONNX

47

48

```commandline

49

pip install transformers onnxruntime huggingface_hub

50

```

51

52

```python

53

from huggingface_hub import hf_hub_download

54

from transformers import AutoTokenizer, AutoConfig

55

import onnxruntime as ort

56

57

model_id = "gravitee-io/bert-small-pii-detection"

58

59

tokenizer = AutoTokenizer.from_pretrained(model_id)

60

id2label = AutoConfig.from_pretrained(model_id).id2label

61

session = ort.InferenceSession(hf_hub_download(model_id, "model.quant.onnx"))

62

63

text = "Contact John Smith at john@example.com"

64

enc = tokenizer(text, return_tensors="np")

65

inputs = {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]}

66

logits = session.run(None, inputs)[0][0]

67

68

tokens = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])

69

labels = [id2label[i] for i in logits.argmax(-1)]

70

71

for tok, label in zip(tokens, labels):

72

print(f"{tok:<20} {label}")

73

```

74

75

## Intended use

76

77

Detect personally identifiable information (PII) spans in `english` text. Suitable

78

for privacy filtering, redaction pipelines, and data-leak prevention particularly on

79

structured data (JSON, HTML, XML, SQL, Document)

80

81

## Evaluation

82

83

| Metric | Value |

84

|---|---|

85

| F1 | 0.8686 |

86

| Precision | 0.8182 |

87

| Recall | 0.9256 |

88

| Eval loss | 0.0132 |

89

90

91

## Limitations

92

93

* English-focused; other languages will degrade

94

* Domain drift is real: audit on your own data

95

---

96

97

## Benchmarks

98

99

External-corpus evaluation (English only), seqeval. Last run: 2026-05-21.

100

101

| Benchmark                                          | Examples | FP32 micro F1 | FP32 macro F1 | INT8 micro F1 | INT8 macro F1 |

102

|----------------------------------------------------|---:|---:|---:|---:|---:|

103

| `gretelai/gretel-pii-masking-en-v1:test` | 5,000 | 0.9141 | 0.8971 | 0.9121 | 0.8860 |

104

| `gretelai/synthetic_pii_finance_multilingual:test` | 2,962 | 0.7534 | 0.7354 | 0.7498 | 0.7351 |

105

| `DataikuNLP/kiji-pii-training-data:test` | 1,033 | 0.9259 | 0.8685 | 0.9265 | 0.8725 |

106

| `beki/privy:test` | 28,843 | 0.8809 | 0.9694 | 0.8800 | 0.9680 |

107

| `beki/privy:test-large` | 120,574 | 0.9833 | 0.9810 | 0.9825 | 0.9801 |

108

109

### Per-entity breakdown

110

111

<details>

112

<summary><code>gretelai/gretel-pii-masking-en-v1:test</code></summary>

113

114

115

|---|---:|---|---:|---:|---|

116

| `AGE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

117

| `COORDINATE` | 0.8966 | 0.876 / 0.918 | 85 | 0.8966 | 0.876 / 0.918 |

118

| `CREDIT_CARD` | 0.9572 | 0.937 / 0.979 | 663 | 0.9524 | 0.926 / 0.980 |

119

| `DATE_TIME` | 0.9605 | 0.935 / 0.988 | 3,805 | 0.9568 | 0.929 / 0.987 |

120

| `EMAIL_ADDRESS` | 0.9854 | 0.976 / 0.995 | 1,048 | 0.9854 | 0.976 / 0.995 |

121

| `FINANCIAL` | 0.7143 | 0.641 / 0.806 | 31 | 0.6857 | 0.615 / 0.774 |

122

| `IMEI` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

123

| `IP_ADDRESS` | 0.9819 | 0.974 / 0.990 | 961 | 0.9829 | 0.976 / 0.990 |

124

| `LOCATION` | 0.8549 | 0.853 / 0.857 | 1,760 | 0.8561 | 0.855 / 0.857 |

125

| `NRP` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

126

| `ORGANIZATION` | 0.7159 | 0.611 / 0.865 | 185 | 0.6974 | 0.587 / 0.859 |

127

| `PASSWORD` | 0.8712 | 0.793 / 0.966 | 119 | 0.8679 | 0.788 / 0.966 |

128

| `PERSON` | 0.7973 | 0.781 / 0.814 | 3,209 | 0.7948 | 0.781 / 0.809 |

129

| `PHONE_NUMBER` | 0.9738 | 0.962 / 0.986 | 904 | 0.9701 | 0.955 / 0.986 |

130

| `TITLE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

131

| `URL` | 0.8846 | 0.793 / 1.000 | 23 | 0.8302 | 0.733 / 0.957 |

132

| `US_BANK_NUMBER` | 0.9610 | 0.962 / 0.960 | 398 | 0.9611 | 0.960 / 0.962 |

133

| `US_DRIVER_LICENSE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

134

| `US_ITIN` | 0.8936 | 0.875 / 0.913 | 23 | 0.8333 | 0.800 / 0.870 |

135

| `US_LICENSE_PLATE` | 0.9171 | 0.873 / 0.965 | 579 | 0.9156 | 0.871 / 0.965 |

136

| `US_PASSPORT` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

137

| `US_SSN` | 0.9880 | 0.985 / 0.991 | 1,705 | 0.9898 | 0.988 / 0.992 |

138

139

</details>

140

141

<details>

142

<summary><code>gretelai/synthetic_pii_finance_multilingual:test</code></summary>

143

144

145

|---|---:|---|---:|---:|---|

146

| `AGE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

147

| `COORDINATE` | 0.6000 | 0.483 / 0.792 | 53 | 0.6087 | 0.494 / 0.792 |

148

| `CREDIT_CARD` | 0.5874 | 0.467 / 0.792 | 53 | 0.6143 | 0.494 / 0.811 |

149

| `DATE_TIME` | 0.7410 | 0.667 / 0.833 | 4,294 | 0.7406 | 0.667 / 0.833 |

150

| `EMAIL_ADDRESS` | 0.7971 | 0.746 / 0.856 | 576 | 0.7981 | 0.741 / 0.865 |

151

| `FINANCIAL` | 0.7048 | 0.632 / 0.796 | 294 | 0.6967 | 0.624 / 0.789 |

152

| `IBAN_CODE` | 0.8514 | 0.778 / 0.940 | 67 | 0.8571 | 0.787 / 0.940 |

153

| `IP_ADDRESS` | 0.7854 | 0.796 / 0.775 | 111 | 0.7892 | 0.786 / 0.793 |

154

| `LOCATION` | 0.7554 | 0.684 / 0.844 | 1,938 | 0.7506 | 0.677 / 0.842 |

155

| `NRP` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

156

| `ORGANIZATION` | 0.6975 | 0.612 / 0.811 | 2,702 | 0.6876 | 0.602 / 0.802 |

157

| `PASSWORD` | 0.6392 | 0.508 / 0.861 | 36 | 0.5941 | 0.462 / 0.833 |

158

| `PERSON` | 0.8125 | 0.778 / 0.851 | 3,295 | 0.8085 | 0.771 / 0.850 |

159

| `PHONE_NUMBER` | 0.8648 | 0.791 / 0.953 | 406 | 0.8651 | 0.790 / 0.956 |

160

| `TITLE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

161

| `URL` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

162

| `US_BANK_NUMBER` | 0.6038 | 0.511 / 0.738 | 65 | 0.5976 | 0.495 / 0.754 |

163

| `US_DRIVER_LICENSE` | 0.7731 | 0.697 / 0.868 | 53 | 0.7797 | 0.708 / 0.868 |

164

| `US_ITIN` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

165

| `US_LICENSE_PLATE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

166

| `US_PASSPORT` | 0.7419 | 0.708 / 0.780 | 59 | 0.7680 | 0.727 / 0.814 |

167

| `US_SSN` | 0.8112 | 0.773 / 0.853 | 68 | 0.8056 | 0.763 / 0.853 |

168

169

</details>

170

171

<details>

172

<summary><code>DataikuNLP/kiji-pii-training-data:test</code></summary>

173

174

175

|---|---:|---|---:|---:|---|

176

| `AGE` | 0.8682 | 0.789 / 0.966 | 116 | 0.8794 | 0.801 / 0.974 |

177

| `CREDIT_CARD` | 0.9431 | 0.892 / 1.000 | 58 | 0.9587 | 0.921 / 1.000 |

178

| `DATE_TIME` | 0.8276 | 0.742 / 0.936 | 141 | 0.8354 | 0.754 / 0.936 |

179

| `EMAIL_ADDRESS` | 0.9942 | 0.989 / 1.000 | 258 | 0.9942 | 0.989 / 1.000 |

180

| `IBAN_CODE` | 0.9655 | 0.942 / 0.990 | 99 | 0.9703 | 0.951 / 0.990 |

181

| `LOCATION` | 0.9115 | 0.878 / 0.948 | 3,630 | 0.9116 | 0.881 / 0.945 |

182

| `ORGANIZATION` | 0.7439 | 0.716 / 0.774 | 274 | 0.7435 | 0.712 / 0.777 |

183

| `PASSWORD` | 0.8732 | 0.845 / 0.903 | 103 | 0.9005 | 0.880 / 0.922 |

184

| `PERSON` | 0.9685 | 0.956 / 0.981 | 1,987 | 0.9665 | 0.952 / 0.981 |

185

| `PHONE_NUMBER` | 0.9676 | 0.968 / 0.968 | 247 | 0.9676 | 0.968 / 0.968 |

186

| `TITLE` | 0.0000 | 0.000 / 0.000 | 3 | 0.0000 | 0.000 / 0.000 |

187

| `URL` | 0.9474 | 0.936 / 0.959 | 169 | 0.9419 | 0.926 / 0.959 |

188

| `US_DRIVER_LICENSE` | 0.9323 | 0.900 / 0.967 | 121 | 0.9558 | 0.930 / 0.983 |

189

| `US_ITIN` | 0.9474 | 0.947 / 0.947 | 95 | 0.9474 | 0.947 / 0.947 |

190

| `US_LICENSE_PLATE` | 0.9669 | 0.959 / 0.975 | 120 | 0.9508 | 0.935 / 0.967 |

191

| `US_PASSPORT` | 0.9787 | 0.966 / 0.991 | 116 | 0.9746 | 0.958 / 0.991 |

192

| `US_SSN` | 0.9291 | 0.892 / 0.969 | 196 | 0.9337 | 0.900 / 0.969 |

193

194

</details>

195

196

<details>

197

<summary><code>beki/privy:test</code></summary>

198

199

200

|---|---:|---|---:|---:|---|

201

| `AGE` | 0.9659 | 0.934 / 1.000 | 764 | 0.9610 | 0.926 / 0.999 |

202

| `COORDINATE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

203

| `CREDIT_CARD` | 1.0000 | 1.000 / 1.000 | 757 | 1.0000 | 1.000 / 1.000 |

204

| `DATE_TIME` | 0.9975 | 0.995 / 1.000 | 5,289 | 0.9975 | 0.995 / 0.999 |

205

| `EMAIL_ADDRESS` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

206

| `FINANCIAL` | 0.9584 | 0.924 / 0.996 | 2,243 | 0.9541 | 0.916 / 0.996 |

207

| `HONORIFIC` | 0.9970 | 0.994 / 1.000 | 2,345 | 0.9972 | 0.995 / 1.000 |

208

| `IBAN_CODE` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

209

| `IMEI` | 1.0000 | 1.000 / 1.000 | 769 | 0.9994 | 0.999 / 1.000 |

210

| `IP_ADDRESS` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

211

| `LOCATION` | 0.8851 | 0.968 / 0.815 | 12,930 | 0.8850 | 0.968 / 0.815 |

212

| `MAC_ADDRESS` | 0.9986 | 0.997 / 1.000 | 735 | 0.9959 | 0.992 / 1.000 |

213

| `NRP` | 0.9958 | 0.992 / 0.999 | 3,829 | 0.9956 | 0.992 / 0.999 |

214

| `ORGANIZATION` | 0.9820 | 0.977 / 0.987 | 1,493 | 0.9807 | 0.974 / 0.987 |

215

| `PASSWORD` | 0.9348 | 0.881 / 0.996 | 720 | 0.9386 | 0.886 / 0.997 |

216

| `PERSON` | 0.9897 | 0.988 / 0.991 | 7,986 | 0.9878 | 0.986 / 0.990 |

217

| `PHONE_NUMBER` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

218

| `TITLE` | 0.9661 | 0.942 / 0.992 | 732 | 0.9655 | 0.939 / 0.993 |

219

| `URL` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

220

| `US_BANK_NUMBER` | 0.9951 | 0.990 / 1.000 | 717 | 0.9958 | 0.992 / 1.000 |

221

| `US_DRIVER_LICENSE` | 0.9303 | 0.890 / 0.974 | 781 | 0.9225 | 0.875 / 0.976 |

222

| `US_ITIN` | 0.9811 | 0.965 / 0.997 | 754 | 0.9824 | 0.968 / 0.997 |

223

| `US_LICENSE_PLATE` | 0.9390 | 0.895 / 0.987 | 788 | 0.9334 | 0.885 / 0.987 |

224

| `US_PASSPORT` | 0.9334 | 0.893 / 0.977 | 753 | 0.9320 | 0.894 / 0.973 |

225

| `US_SSN` | 0.0000 | 0.000 / 0.000 | 0 | 0.0000 | 0.000 / 0.000 |

226

227

</details>

228

229

<details>

230

<summary><code>beki/privy:test-large</code></summary>

231

232

233

|---|---:|---|---:|---:|---|

234

| `AGE` | 0.9447 | 0.895 / 1.000 | 3,092 | 0.9441 | 0.895 / 0.999 |

235

| `COORDINATE` | 0.9994 | 0.999 / 1.000 | 9,543 | 0.9996 | 0.999 / 1.000 |

236

| `CREDIT_CARD` | 0.9968 | 0.997 / 0.996 | 3,151 | 0.9970 | 0.997 / 0.997 |

237

| `DATE_TIME` | 0.9925 | 0.986 / 1.000 | 22,136 | 0.9923 | 0.985 / 0.999 |

238

| `EMAIL_ADDRESS` | 0.9992 | 0.999 / 1.000 | 3,142 | 0.9987 | 0.998 / 1.000 |

239

| `FINANCIAL` | 0.9481 | 0.907 / 0.993 | 9,360 | 0.9433 | 0.898 / 0.993 |

240

| `HONORIFIC` | 0.9982 | 0.997 / 1.000 | 9,584 | 0.9982 | 0.997 / 1.000 |

241

| `IBAN_CODE` | 0.9982 | 0.996 / 1.000 | 3,099 | 0.9982 | 0.996 / 1.000 |

242

| `IMEI` | 0.9998 | 1.000 / 1.000 | 3,116 | 0.9997 | 0.999 / 1.000 |

243

| `IP_ADDRESS` | 0.9972 | 0.994 / 1.000 | 3,185 | 0.9970 | 0.995 / 0.999 |

244

| `LOCATION` | 0.9764 | 0.964 / 0.990 | 43,932 | 0.9761 | 0.963 / 0.989 |

245

| `MAC_ADDRESS` | 0.9957 | 0.992 / 1.000 | 3,137 | 0.9951 | 0.991 / 1.000 |

246

| `NRP` | 0.9948 | 0.991 / 0.999 | 15,943 | 0.9948 | 0.991 / 0.998 |

247

| `ORGANIZATION` | 0.9794 | 0.970 / 0.989 | 6,165 | 0.9762 | 0.963 / 0.989 |

248

| `PASSWORD` | 0.9656 | 0.936 / 0.997 | 3,082 | 0.9599 | 0.925 / 0.997 |

249

| `PERSON` | 0.9887 | 0.987 / 0.990 | 32,380 | 0.9878 | 0.985 / 0.990 |

250

| `PHONE_NUMBER` | 0.9979 | 0.996 / 1.000 | 3,099 | 0.9974 | 0.995 / 1.000 |

251

| `TITLE` | 0.9744 | 0.954 / 0.995 | 3,192 | 0.9696 | 0.945 / 0.996 |

252

| `URL` | 0.9985 | 0.997 / 1.000 | 6,237 | 0.9985 | 0.997 / 1.000 |

253

| `US_BANK_NUMBER` | 0.9948 | 0.991 / 0.999 | 3,091 | 0.9937 | 0.989 / 0.998 |

254

| `US_DRIVER_LICENSE` | 0.9238 | 0.874 / 0.979 | 3,041 | 0.9208 | 0.869 / 0.979 |

255

| `US_ITIN` | 0.9821 | 0.966 / 0.999 | 2,995 | 0.9829 | 0.967 / 0.999 |

256

| `US_LICENSE_PLATE` | 0.9458 | 0.902 / 0.994 | 3,049 | 0.9414 | 0.894 / 0.994 |

257

| `US_PASSPORT` | 0.9344 | 0.889 / 0.985 | 3,044 | 0.9405 | 0.901 / 0.983 |

258

| `US_SSN` | 0.9982 | 0.996 / 1.000 | 2,980 | 0.9990 | 0.998 / 1.000 |

259

260

</details>

261

262

## Citation

263

264

Data citation are present in the dataset card used for this model.

265

If you use the model, please consider citing the papers:

266

267

```

268

@misc{bhargava2021generalization,

269

title={Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics},

270

author={Prajjwal Bhargava and Aleksandr Drozd and Anna Rogers},

271

year={2021},

272

eprint={2110.01518},

273

archivePrefix={arXiv},

274

primaryClass={cs.CL}

275

}

276

277

@article{DBLP:journals/corr/abs-1908-08962,

278

author = {Iulia Turc and

279

Ming{-}Wei Chang and

280

Kenton Lee and

281

Kristina Toutanova},

282

title = {Well-Read Students Learn Better: The Impact of Student Initialization

283

on Knowledge Distillation},

284

journal = {CoRR},

285

volume = {abs/1908.08962},

286

year = {2019},

287

url = {http://arxiv.org/abs/1908.08962},

288

eprinttype = {arXiv},

289

eprint = {1908.08962},

290

timestamp = {Thu, 29 Aug 2019 16:32:34 +0200},

291

biburl = {https://dblp.org/rec/journals/corr/abs-1908-08962.bib},

292

bibsource = {dblp computer science bibliography, https://dblp.org}

293

}

294

```