README.md · bytkim_Qwen3.8-27B-pi-GGUF

1

---

2

quantized_by: bartowski

3

pipeline_tag: image-text-to-text

4

license: apache-2.0

5

base_model: bytkim/Qwen3.8-27B-pi

6

tags:

7

- mtp

8

- multi-token-prediction

9

- speculative-decoding

10

- vision

11

- image

12

- multimodal

13

- image-text-to-text

14

- text-generation-inference

15

- qwen

16

- qwen3.8

17

- qwen38

18

- 27b

19

- pi

20

- coding

21

- coder

22

- code-generation

23

- agent

24

- tool-use

25

- tool-calling

26

- function-calling

27

- reasoning

28

- conversational

29

- sft

30

- grpo

31

- reinforcement-learning

32

- bf16

33

base_model_relation: quantized

34

---

35

36

## Llamacpp imatrix Quantizations of Qwen3.8-27B-pi by bytkim

37

38

Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b11279">b11279</a> for quantization.

39

40

Original model: https://huggingface.co/bytkim/Qwen3.8-27B-pi

41

42

**Model details:**

43

- Parameter count: 28B (source checkpoint)

44

- Input support: text, image (with mmproj file) - [details](#multimodal)

45

- Speculative decoding: yes (MTP) - [details](#mtp)

46

- imatrix: yes - [details](#imatrix)

47

48

[How to run](#how-to-run)

49

50

## Prompt format

51

52

```

53

<|im_start|>system

54

Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.

55

56

{system_prompt}<|im_end|>

57

<|im_start|>user

58

{prompt}<|im_end|>

59

<|im_start|>assistant

60

<think>

61

```

62

63

<details><summary>Prompt format with tool definitions</summary>

64

65

```

66

<|im_start|>system

67

Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.

68

69

# Tools

70

71

You have access to the following functions:

72

73

<tools>

74

{"type": "function", "function": {"name": "get_stock_price", "description": "Get the current stock price", "parameters": {"type": "object", "properties": {"symbol": {"type": "string", "description": "The stock symbol, e.g. AAPL, GOOG"}}, "required": ["symbol"]}}}

75

</tools>

76

77

If you choose to call a function ONLY reply in the following format with NO suffix:

78

79

<tool_call>

80

<function=example_function_name>

81

<parameter=example_parameter_1>

82

value_1

83

</parameter>

84

<parameter=example_parameter_2>

85

This is the value for the second parameter

86

that can span

87

multiple lines

88

</parameter>

89

</function>

90

</tool_call>

91

92

<IMPORTANT>

93

Reminder:

94

- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags

95

- Required parameters MUST be specified

96

- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after

97

- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls

98

</IMPORTANT>

99

100

{system_prompt}<|im_end|>

101

<|im_start|>user

102

{prompt}<|im_end|>

103

<|im_start|>assistant

104

<think>

105

```

106

107

</details>

108

109

**Don't know which to choose?** Grab [Q4_K_M](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q4_K_M.gguf) (17.44GB) - usually a good mix of size and performance. Download instructions available [here](#downloading-using-the-hugging-face-cli)

110

111

## Available files:

112

113

114

| -------- | ---------- | --------- | ----- | ----------- |

115

| [bytkim_Qwen3.8-27B-pi-bf16.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/tree/main/bytkim_Qwen3.8-27B-pi-bf16) | bf16 | 54.66GB | true | Full BF16 weights. |

116

| [bytkim_Qwen3.8-27B-pi-Q8_0.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q8_0.gguf) | Q8_0 | 29.12GB | false | Extremely high quality, generally unneeded but max available quant. |

117

| [bytkim_Qwen3.8-27B-pi-Q6_K_L.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q6_K_L.gguf) | Q6_K_L | 24.96GB | false | The large size of Q6_K, about halfway to Q8_0 in size. Very high quality, near perfect, *recommended*. |

118

| [bytkim_Qwen3.8-27B-pi-Q6_K.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q6_K.gguf) | Q6_K | 23.86GB | false | Very high quality, near perfect, *recommended*. |

119

| [bytkim_Qwen3.8-27B-pi-Q6_K_S.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q6_K_S.gguf) | Q6_K_S | 22.86GB | false | Very high quality, near perfect, a little smaller than Q6_K with almost all of the model at Q6_K precision, *recommended*. |

120

| [bytkim_Qwen3.8-27B-pi-Q5_K_M.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q5_K_M.gguf) | Q5_K_M | 20.92GB | false | High quality, *recommended*. |

121

| [bytkim_Qwen3.8-27B-pi-Q5_K_S.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q5_K_S.gguf) | Q5_K_S | 19.57GB | false | High quality, *recommended*. |

122

| [bytkim_Qwen3.8-27B-pi-Q4_K_L.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q4_K_L.gguf) | Q4_K_L | 18.82GB | false | The large size of Q4_K, between Q4_K_M and Q5_K_S: more of the most sensitive weights kept at higher precision, *recommended*. |

123

| [bytkim_Qwen3.8-27B-pi-Q4_1.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q4_1.gguf) | Q4_1 | 17.83GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |

124

| [bytkim_Qwen3.8-27B-pi-Q4_K_M.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q4_K_M.gguf) | Q4_K_M | 17.44GB | false | Good quality, default size for most use cases, *recommended*. |

125

| [bytkim_Qwen3.8-27B-pi-IQ4_NL.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ4_NL.gguf) | IQ4_NL | 17.44GB | false | Similar to IQ4_XS, but slightly larger. |

126

| [bytkim_Qwen3.8-27B-pi-Q4_K_S.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q4_K_S.gguf) | Q4_K_S | 16.36GB | false | Slightly lower quality with more space savings, *recommended*. |

127

| [bytkim_Qwen3.8-27B-pi-Q4_0.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q4_0.gguf) | Q4_0 | 16.35GB | false | Legacy format, kept for compatibility with older tools. |

128

| [bytkim_Qwen3.8-27B-pi-IQ4_XS.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ4_XS.gguf) | IQ4_XS | 15.48GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |

129

| [bytkim_Qwen3.8-27B-pi-IQ3_M.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ3_M.gguf) | IQ3_M | 14.86GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |

130

| [bytkim_Qwen3.8-27B-pi-Q3_K_L.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q3_K_L.gguf) | Q3_K_L | 14.12GB | false | Lower quality but usable, good for low RAM availability. |

131

| [bytkim_Qwen3.8-27B-pi-Q3_K_M.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q3_K_M.gguf) | Q3_K_M | 13.40GB | false | Low quality. |

132

| [bytkim_Qwen3.8-27B-pi-IQ3_XS.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ3_XS.gguf) | IQ3_XS | 12.80GB | false | Lower quality, new method with decent performance, slightly better than Q3_K_S. |

133

| [bytkim_Qwen3.8-27B-pi-Q3_K_S.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q3_K_S.gguf) | Q3_K_S | 12.74GB | false | Low quality, *not* recommended. |

134

| [bytkim_Qwen3.8-27B-pi-IQ3_XXS.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ3_XXS.gguf) | IQ3_XXS | 12.32GB | false | Lower quality, new method with decent performance, comparable to Q3 quants. |

135

| [bytkim_Qwen3.8-27B-pi-Q2_K.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-Q2_K.gguf) | Q2_K | 10.82GB | false | Very low quality but surprisingly usable. |

136

| [bytkim_Qwen3.8-27B-pi-IQ2_M.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ2_M.gguf) | IQ2_M | 10.52GB | false | Relatively low quality, uses SOTA techniques to be surprisingly usable. |

137

| [bytkim_Qwen3.8-27B-pi-IQ2_S.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ2_S.gguf) | IQ2_S | 9.68GB | false | Low quality, uses SOTA techniques to be usable. |

138

| [bytkim_Qwen3.8-27B-pi-IQ2_XS.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ2_XS.gguf) | IQ2_XS | 9.09GB | false | Low quality, uses SOTA techniques to be usable. |

139

| [bytkim_Qwen3.8-27B-pi-IQ2_XXS.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-IQ2_XXS.gguf) | IQ2_XXS | 8.88GB | false | Very low quality, uses SOTA techniques to be usable. |

140

141

Download a specific file:

142

143

```

144

hf download bartowski/bytkim_Qwen3.8-27B-pi-GGUF --include "bytkim_Qwen3.8-27B-pi-Q4_K_M.gguf" --local-dir ./

145

```

146

147

## Downloading using the Hugging Face CLI

148

149

<details>

150

<summary>Click to view download instructions</summary>

151

152

First, make sure you have the Hugging Face CLI installed:

153

154

```

155

pip install -U "huggingface_hub[cli]"

156

```

157

158

Download a specific file:

159

160

```

161

hf download bartowski/bytkim_Qwen3.8-27B-pi-GGUF --include "bytkim_Qwen3.8-27B-pi-Q4_K_M.gguf" --local-dir ./

162

```

163

164

The files marked `true` in the Split column above are stored as multiple parts in a folder. To download all the parts to a local folder, run:

165

166

```

167

hf download bartowski/bytkim_Qwen3.8-27B-pi-GGUF --include "bytkim_Qwen3.8-27B-pi-bf16/*" --local-dir ./

168

```

169

170

You can either specify a new local-dir (bytkim_Qwen3.8-27B-pi-bf16) or download them all in place (./)

171

172

</details>

173

174

## How to run

175

176

These quants run with [llama.cpp](https://github.com/ggml-org/llama.cpp) - installable in one line via [llama.app](https://llama.app/):

177

178

```

179

curl -LsSf https://llama.app/install.sh | sh

180

llama-server -hf bartowski/bytkim_Qwen3.8-27B-pi-GGUF:Q4_K_M

181

```

182

183

llama-server includes a built-in chat web UI, served at http://localhost:8080 by default.

184

185

These quants were made with llama.cpp release b11279 - if this model's architecture is newly supported, you'll need that release or newer to run them.

186

187

They also work in: [LM Studio](https://lmstudio.ai/) · [koboldcpp](https://github.com/LostRuins/koboldcpp) · [ramalama](https://github.com/containers/ramalama) · [Jan AI](https://www.jan.ai/) · [Text Generation Web UI](https://github.com/oobabooga/text-generation-webui) · [LoLLMs](https://github.com/ParisNeo/lollms) · [Atomic Chat](https://atomic.chat/)

188

189

## Multimodal

190

191

This model supports image input. Alongside the quants, this repo includes the multimodal projector files [mmproj-bytkim_Qwen3.8-27B-pi-bf16.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/mmproj-bytkim_Qwen3.8-27B-pi-bf16.gguf) and [mmproj-bytkim_Qwen3.8-27B-pi-f16.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/mmproj-bytkim_Qwen3.8-27B-pi-f16.gguf), which pair with any quant above.

192

193

llama.cpp downloads the mmproj automatically when using `-hf` as shown above; if you're loading files manually, pass it with `--mmproj`.

194

195

## MTP

196

197

This model has MTP (Multi-Token Prediction) layers, and they are included in these quants

198

199

MTP layers act as a built-in draft model, letting llama.cpp run speculative decoding for faster generation. To use them, add the following flag to your llama.cpp command:

200

201

```

202

--spec-type draft-mtp

203

```

204

205

Note: the MTP layers are stored at Q4_0 in every quant except Q8_0. Imatrix calibration does not exercise them, so the imatrix has nothing to offer these layers, and Q4_0 is chosen for its speed, which massively benefits MTP performance.

206

207

## Per-tensor layouts

208

209

Some of these files were built with a layout computed for this model instead of llama.cpp's standard one-size-fits-all rules. A Q4_K_M is still mostly Q4_K; the extra precision goes to the weights this particular model is most sensitive to. The S, M or L in a name says how much of the model stays at the base precision: about 90 % for S, 70 % for M and 50 % for L. An `_L` name is simply the large size of its family. Q4_K_L is to Q4_K_M what Q4_K_M is to Q4_K_S; Q6_K_S, Q6_K and Q6_K_L are the small, medium and large sizes of Q6_K, with Q6_K_L about halfway to Q8_0. In earlier releases an `_L` name meant the embedding and output weights were kept at Q8_0; in these files it means the larger size of the base type. There is no size target, so each file's bits per weight is reported rather than promised.

210

211

The layout each of these files was built with is published in the [`layouts/`](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/tree/main/layouts) folder: `<file>.tensor-types.txt` is the exact `--tensor-type-file` given to `llama-quantize`, and `<file>.layout.json` records how it was computed, including the generator version, the llama.cpp release and the commit, so any of them can be rebuilt. The code that computed them is public at [`quantization-config`](https://github.com/bartowski1182/quantization-config); its tag `key-3bf8b43e20a68f0b` is the exact snapshot these files record. The method is described in [this write-up](https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization).

212

213

Checked on [bartowski/Altworld_Hemmingway-1-GGUF](https://huggingface.co/bartowski/Altworld_Hemmingway-1-GGUF) (same architecture and tensor shapes) before any of these files were released: Q4_K_M reached 0.91×, Q3_K_M 0.76× and IQ2_XXS 0.76× the KL divergence of the standard layout at the same file size.

214

215

<details>

216

<summary>Layout details</summary>

217

218

Files built from a computed layout:

219

220

221

| ----- | ---- | ---------------- | ---------------- | ---------------------- |

222

| Q6_K_L | 24.96GB | 7.41 | 7.31 | 50 % |

223

| Q6_K | 23.86GB | 7.04 | 6.99 | 70 % |

224

| Q6_K_S | 22.86GB | 6.71 | 6.69 | 90 % |

225

| Q5_K_M | 20.92GB | 6.13 | 6.13 | 70 % |

226

| Q5_K_S | 19.57GB | 5.69 | 5.73 | 90 % |

227

| Q4_K_L | 18.82GB | 5.49 | 5.51 | 50 % |

228

| Q4_K_M | 17.44GB | 5.04 | 5.11 | 70 % |

229

| IQ4_NL | 17.44GB | 5.04 | 5.11 | 70 % |

230

| Q4_K_S | 16.36GB | 4.69 | 4.79 | 90 % |

231

| IQ4_XS | 15.48GB | 4.41 | 4.53 | 90 % |

232

| IQ3_M | 14.86GB | 4.25 | 4.35 | 50 % |

233

| Q3_K_L | 14.12GB | 4.00 | 4.14 | 50 % |

234

| Q3_K_M | 13.40GB | 3.77 | 3.92 | 70 % |

235

| IQ3_XS | 12.80GB | 3.57 | 3.75 | 90 % |

236

| Q3_K_S | 12.74GB | 3.55 | 3.73 | 90 % |

237

| IQ3_XXS | 12.32GB | 3.41 | 3.61 | 70 % |

238

| Q2_K | 10.82GB | 2.98 | 3.17 | 70 % |

239

| IQ2_M | 10.52GB | 2.88 | 3.08 | 70 % |

240

| IQ2_S | 9.68GB | 2.60 | 2.84 | 70 % |

241

| IQ2_XS | 9.09GB | 2.40 | 2.66 | 90 % |

242

| IQ2_XXS | 8.88GB | 2.34 | 2.60 | 70 % |

243

244

Checked on [bartowski/Altworld_Hemmingway-1-GGUF](https://huggingface.co/bartowski/Altworld_Hemmingway-1-GGUF), which shares this model's architecture and tensor shapes: the computed layout against the standard one, measured by KL divergence against the unquantized model. The ratio compares each computed file with the standard ladder read at that file's own size, so it can differ from the two KLD columns when the two files differ in size.

245

246

247

| ----- | ------------------- | ------------------- | ------------------- | --------------------- |

248

| Q4_K_M | 0.0105 ± 0.0002 | 0.0100 ± 0.0002 | 0.91× | −1.9 % |

249

| Q3_K_M | 0.0444 ± 0.0006 | 0.0370 ± 0.0005 | 0.76× | −8.2 % |

250

| IQ2_XXS | 0.2391 ± 0.0026 | 0.2609 ± 0.0028 | 0.76× | −5.4 % |

251

252

How it works: the base type is a floor for every body tensor and a fixed share of the body bytes stays at it (90 % for S, 70 % for M, 50 % for L); the remaining bytes go where a cross-model sensitivity prior, measured by KL divergence against the unquantized model, says they buy the most quality. The embedding and output tensors are sized by their share of the file: a small table is kept at Q8_0, a large one follows the file's bitrate. A K-quant and the IQ quant with the same base bitrate (Q3_K_S and IQ3_XS, Q3_K_M and IQ3_S, Q3_K_L and IQ3_M) come out at about the same size; the IQ file is the GPU-oriented twin.

253

254

</details>

255

256

## imatrix

257

258

All quants made using imatrix option, with a calibration corpus rendered through this model's own chat template. The corpus pairs plain prose with tool-calling and reasoning conversations ([corpus source data](https://gist.github.com/bartowski1182/e26453c0404e24eb317543ec5360f87a)), encoded exactly as this model sees them at inference and processed with `--parse-special`, so chat-format special tokens contribute to the importance matrix. The corpus rendered for this model is included in this repo: [bytkim_Qwen3.8-27B-pi-calibration-v6.txt](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-calibration-v6.txt). The imatrix is available here: [bytkim_Qwen3.8-27B-pi-imatrix.gguf](https://huggingface.co/bartowski/bytkim_Qwen3.8-27B-pi-GGUF/blob/main/bytkim_Qwen3.8-27B-pi-imatrix.gguf).

259

260

<details>

261

<summary>Calibration render details</summary>

262

263

```json

264

{

265

"recipe": "calibration-v6",

266

"encoder": "chat_template",

267

"library_versions": {

268

"transformers": "5.9.0",

269

"tokenizers": "0.22.2",

270

"tiktoken": "0.14.0",

271

"blobfile": "3.3.0",

272

"huggingface_hub": "1.31.0"

273

},

274

"chunk_size": 512,

275

"prose_chunks": 214,

276

"tool_chunks": 369,

277

"total_chunks": 583,

278

"n_conversations": 137,

279

"conversation_token_lengths": [

280

604,

281

1667,

282

1297,

283

1559,

284

1185,

285

1404,

286

3220,

287

866,

288

1267,

289

1462,

290

1107,

291

2136,

292

921,

293

1293,

294

2829,

295

1313,

296

1160,

297

1026,

298

777,

299

758,

300

1402,

301

1098,

302

1447,

303

1242,

304

1912,

305

1538,

306

1694,

307

948,

308

1455,

309

1686,

310

1609,

311

1260,

312

1308,

313

1077,

314

1060,

315

1752,

316

1691,

317

1215,

318

517,

319

1950,

320

1451,

321

1166,

322

1438,

323

2042,

324

2130,

325

1355,

326

1649,

327

947,

328

2996,

329

1142,

330

2908,

331

827,

332

1064,

333

1010,

334

947,

335

738,

336

2530,

337

955,

338

1188,

339

1127,

340

1271,

341

1211,

342

966,

343

1259,

344

1223,

345

1628,

346

953,

347

1592,

348

2166,

349

881,

350

365,

351

1162,

352

3375,

353

2876,

354

749,

355

982,

356

1072,

357

1113,

358

1338,

359

1122,

360

1191,

361

835,

362

1250,

363

1111,

364

1331,

365

1586,

366

1443,

367

2106,

368

913,

369

698,

370

2806,

371

682,

372

1443,

373

1715,

374

1987,

375

1243,

376

683,

377

1396,

378

1168,

379

1769,

380

1884,

381

1755,

382

857,

383

1055,

384

1077,

385

2896,

386

780,

387

774,

388

803,

389

1452,

390

1092,

391

1617,

392

808,

393

392,

394

361,

395

2666,

396

1058,

397

1179,

398

1870,

399

2079,

400

2735,

401

2717,

402

856,

403

1020,

404

880,

405

999,

406

1279,

407

1039,

408

882,

409

1397,

410

870,

411

750,

412

1783,

413

1067,

414

957,

415

1356,

416

1519

417

]

418

}

419

```

420

421

</details>

422

423

## ARM/AVX information

424

425

llama.cpp automatically "repacks" weights into an interleaved layout at load time for faster inference on ARM and AVX machines - details in [this PR](https://github.com/ggml-org/llama.cpp/pull/9921). This once required downloading special Q4_0_4_4/4_8/8_8 files; those are long gone. Online repacking now covers Q4_0, IQ4_NL, and most K-quants, so no special quant choice is needed for CPU inference.

426

427

## Which file should I choose?

428

429

<details>

430

<summary>Click here for details</summary>

431

432

An older (early 2024) but still useful write-up with charts comparing quant performances is provided by Artefact2 [here](https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9)

433

434

The first thing to figure out is how big a model you can run. To do this, you'll need to figure out how much RAM and/or VRAM you have.

435

436

If you want your model running as FAST as possible, you'll want to fit the whole thing on your GPU's VRAM. Aim for a quant with a file size 1-2GB smaller than your GPU's total VRAM.

437

438

If you want the absolute maximum quality, add both your system RAM and your GPU's VRAM together, then similarly grab a quant with a file size 1-2GB Smaller than that total.

439

440

Hugging Face can also do this math for you: add your hardware in your [Local Apps settings](https://huggingface.co/settings/local-apps) and the model page will show which files fit.

441

442

Next, you'll need to decide if you want to use an 'I-quant' or a 'K-quant'.

443

444

If you don't want to think too much, grab one of the K-quants. These are in format 'QX_K_X', like Q5_K_M.

445

446

If you want to get more into the weeds, you can check out this extremely useful feature chart:

447

448

[llama.cpp feature matrix](https://github.com/ggml-org/llama.cpp/wiki/Feature-matrix)

449

450

But basically, if you're aiming for below Q4, and you're running cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quants. These are in format IQX_X, like IQ3_M. These are newer and offer better performance for their size.

451

452

These I-quants can also be used on CPU, but will be slower than their K-quant equivalent, so speed vs performance is a tradeoff you'll have to decide.

453

454

</details>

455

456

## Credits

457

458

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

459

460

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

461