README.md
3.8 KB · 107 lines · markdown Raw
1 ---
2 library_name: transformers
3 pipeline_tag: text-generation
4 tags:
5 - speculative-decoding
6 - dspark
7 - dflash
8 - specforge
9 - sglang
10 - long-context
11 inference: false
12 ---
13
14 # Kimi K3 DSpark speculator
15
16 ## Overview
17
18 A long-context DSpark speculator for
19 [Kimi K3](https://huggingface.co/moonshotai/Kimi-K3). It supports context lengths of up to 1 million tokens.
20
21 A DSpark speculator for the Kimi K3 target, enabling faster inference through speculative decoding.
22 DSpark extends the DFlash parallel-draft backbone with a Markov logit-bias head and a per-position confidence head. This checkpoint was trained with [SpecForge](https://github.com/sgl-project/SpecForge/) using hidden states from a live [SGLang](https://github.com/sgl-project/sglang) target engine.
23
24
25 ## Model Specifications
26
27 - **Base model:** `moonshotai/Kimi-K3`
28 - **Format:** Safetensors (single-file BF16, 2,249,289,601 parameters)
29 - **Draft:** 5 full-attention Qwen3-style GQA layers, hidden size 7168,
30 64 query heads / 16 KV heads, and `block_size=7`
31 - **Verification width:** 1 current token + 7 draft tokens
32 - **Auxiliary target layers:** `[7, 23, 51, 67, 83]`
33 - **Trained context:** 65,536 tokens
34 - **Target weights:** embedding and unembedding weights are not included
35
36 ## Evaluation Results
37
38 `acc_len` is SGLang's histogram-native request acceptance length, averaged
39 within each question and then equally across questions.
40
41 | Dataset | Questions | acc_len |
42 |---|---:|---:|
43 | SWE-Rebench | 50 | **4.6594** |
44 | GSM8K | 1,319 | **5.4176** |
45 | MATH500 | 500 | **4.1329** |
46 | HumanEval | 164 | **5.5121** |
47 | MBPP | 257 | **5.1980** |
48 | MT-Bench | 80 | **3.9342** |
49 | AIME26 | 30 | **2.9893** |
50 | RULER V2 1M (MK/MV/QA) | 150 (50 per partition) | **4.2553** |
51
52 RULER V2 uses the 1M input configuration. Actual prompts span
53 1,000,432–1,047,925 tokens; partition acc_len is 4.4658 for MK, 4.3081 for
54 MV, and 3.9919 for QA.
55
56 ### AIME26 acc_len by output length
57
58 | Output-token bucket | Questions | Actual output range | acc_len |
59 |---|---:|---:|---:|
60 | 0–1K | 13 | 192–885 | **3.1310** |
61 | 1–2K | 5 | 1,359–1,828 | **2.5773** |
62 | 2–4K | 6 | 2,210–3,732 | **2.5632** |
63 | 4–8K | 4 | 5,187–7,750 | **2.7174** |
64 | 8–16K | 0 | — | — |
65 | 16–32K | 0 | — | — |
66 | 32K+ | 2 | 54,545–224,703 | **4.9194** |
67
68
69 ## Serving with SGLang
70
71 [SGLang Cookbook](https://lmsysorg.mintlify.app/cookbook/autoregressive/Moonshotai/Kimi-K3#hw=b300&pdMode=unified&strategy=low-latency&spec=dspark&hicache=off) provides Kimi K3 deployment recipes.
72 ```bash
73 sglang serve \
74 --trust-remote-code \
75 --model-path moonshotai/Kimi-K3 \
76 --tp-size 8 \
77 --dcp-size 8 \
78 --mem-fraction-static 0.85 \
79 --max-mamba-cache-size 160 \
80 --max-running-requests 32 \
81 --cuda-graph-max-bs-decode 32 \
82 --reasoning-parser kimi_k3 \
83 --tool-call-parser kimi_k3 \
84 --host 0.0.0.0 \
85 --port 30000 \
86 --speculative-algorithm DSPARK \
87 --speculative-draft-model-path RadixArk/Kimi-K3-DSpark \
88 --speculative-dspark-block-size 7 \
89 --speculative-draft-attention-backend trtllm_mha \
90 --enable-linear-replayssm-spec \
91 --context-length 1048576 \
92 --chunked-prefill-size 16384
93 ```
94
95 YaRN-16 is enabled in the published draft config by default with
96 `original_max_position_embeddings=65536` and
97 `max_position_embeddings=1048576`; no separate draft config override is
98 required.
99
100 ## Training Details
101
102 - **Framework:** SpecForge online distillation, with hidden states captured from a frozen Kimi K3 target served by a live SGLang engine. Draft trained from random initialization.
103 - **Loss:** `0.1 CE + 0.9 L1 distillation + 1.0 confidence BCE`, decay gamma 4.0, with 512 sampled anchors per sequence and `block_size=7`.
104 - **Topology:** 4 nodes × 4 GB300 (16 ranks) — 2 × TP8 target replicas, DP2 sampler, FSDP16 `SHARD_GRAD_OP` on the draft, TP-batch scatter. Batch 8 per replica × 32 accumulation steps × 2 replicas = global batch 512.
105
106
107