README.md
3.4 KB · 84 lines · markdown Raw
1 ---
2 license: mit
3 library_name: gguf
4 pipeline_tag: text-generation
5 base_model: deepseek-ai/DeepSeek-V4-Flash
6 base_model_relation: quantized
7 quantized_by: antirez
8 language:
9 - en
10 tags:
11 - gguf
12 - quantized
13 - deepseek
14 - deepseek-v4
15 - deepseek-v4-flash
16 - moe
17 - mixture-of-experts
18 - 2-bit
19 - 4-bit
20 - iq2_xxs
21 - q2_k
22 - q4_k
23 - ds4
24 - apple-silicon
25 - metal
26 ---
27
28 # DeepSeek V4 Flash — GGUF for ds4
29
30 This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).
31
32 https://github.com/antirez/ds4
33
34 ## Files
35
36 | File | Size | Routed experts (`ffn_{gate,up,down}_exps`) | Everything else |
37 |---|---:|---|---|
38 | `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf` | 80.8 GiB | `IQ2_XXS` (gate, up) + `Q2_K` (down) | `Q8_0` attn proj / shared experts / output, `F16` router + embed + indexer + compressor + HC, `F32` norms / sinks / bias |
39 | `DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf` | 153.3 GiB | `Q4_K` (all three) | same as above |
40 | `DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf` | 3.6 GiB | MTP / speculative-decoding support (optional, not standalone). | |
41
42 Use **q2** on 128 GB Mac machines, **q4** on machines with ≥ 256 GB RAM, pair either with **MTP** for optional speculative decoding.
43
44 ## Quantization recipe
45
46 The filename is the spec. In detail, for the **q2** file:
47
48 | Tensor class | Quant | Notes |
49 |---|---|---|
50 | `blk.*.ffn_gate_exps`, `blk.*.ffn_up_exps` | **`IQ2_XXS`** | routed-expert up/gate |
51 | `blk.*.ffn_down_exps` | **`Q2_K`** | routed-expert down (K-quant for quality) |
52 | `blk.*.ffn_{gate,up,down}_shexp` | `Q8_0` | shared experts |
53 | `blk.*.attn_q_a`, `attn_q_b`, `attn_kv`, `attn_output_a`, `attn_output_b` | `Q8_0` | all attention projections (MLA + low-rank output) |
54 | `output.weight` | `Q8_0` | output head |
55 | `token_embd.weight` | `F16` | input embedding |
56 | `blk.*.ffn_gate_inp` (router) | `F16` | learned router |
57 | `blk.*.exp_probs_b` (router bias), `blk.*.attn_sinks`, all `*_norm.weight` | `F32` | |
58 | `blk.*.ffn_gate_tid2eid` | `I32` | hash-routing tables (first 3 layers only) |
59 | `blk.*.attn_compressor_*`, `blk.*.indexer_*`, `blk.*.hc_*`, `blk.*.output_hc_*` | `F16` / `F32` | DSv4-specific auxiliary blocks |
60
61 For the **q4** file, only the three routed-expert classes change to `Q4_K`. Everything else is byte-for-byte identical to the q2 recipe.
62
63 The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on them costs less in average quality than the same treatment of router, projections, or shared experts. Keeping the decision-making components at `Q8_0` preserves model behavior; crushing the experts buys the size.
64
65 ## Usage
66
67 ```bash
68 git clone https://github.com/antirez/ds4
69 cd ds4
70 ./download_model.sh q2 # 128 GB RAM machines
71 ./download_model.sh q4 # >= 256 GB RAM machines
72 ./download_model.sh mtp # optional MTP / speculative decoding
73 make
74
75 ./ds4 -p "Explain Redis streams in one paragraph."
76 ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
77 ```
78
79 The `download_model.sh` script fetches from this repository, resumes partial downloads, and points `./ds4flash.gguf` at the selected variant.
80
81 ## License
82
83 MIT. The base model copyright is held by DeepSeek; the GGUFs are redistributed under the base model's release terms.
84