docs/deploy_guidance.md
3.2 KB · 83 lines · markdown Raw
1 # Kimi-K2.5 Deployment Guide
2
3 > [!Note]
4 > This guide only provides some examples of deployment commands for Kimi-K2.5, which may not be the optimal configuration. Since inference engines are still being updated frequenty, please continue to follow the guidance from their homepage if you want to achieve better inference performance.
5
6 > kimi_k2 reasoning parser and other related features have been merged into vLLM/sglang and will be available in the next release. For now, please use the nightly build Docker image.
7 ## vLLM Deployment
8
9 This model is available in nightly vLLM wheel:
10 ```
11 uv pip install -U vllm \
12 --torch-backend=auto \
13 --extra-index-url https://wheels.vllm.ai/nightly
14 ```
15
16 Here is the example to serve this model on a H200 single node with TP8 via vLLM:
17 ```bash
18 vllm serve $MODEL_PATH -tp 8 --mm-encoder-tp-mode data --trust-remote-code --tool-call-parser kimi_k2 --reasoning-parser kimi_k2
19 ```
20 **Key notes**
21 - `--tool-call-parser kimi_k2`: Required for enabling tool calling
22 - `--reasoning-parser kimi_k2`: Kimi-K2.5 enables thinking mode by default. Make sure to pass this for correct reasoning processing.
23
24 ## SGLang Deployment
25
26 This model is available in SGLang latest main:
27
28 ```
29 pip install "sglang @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"
30 pip install nvidia-cudnn-cu12==9.16.0.29
31 ```
32
33 Similarly, here is the example for it to run with TP8 on H200 in a single node via SGLang:
34 ``` bash
35 sglang serve --model-path $MODEL_PATH --tp 8 --trust-remote-code --tool-call-parser kimi_k2 --reasoning-parser kimi_k2
36 ```
37 **Key parameter notes:**
38 - `--tool-call-parser kimi_k2`: Required when enabling tool usage.
39 - `--reasoning-parser kimi_k2`: Required for correctly processing reasoning content.
40
41 ## KTransformers Deployment
42 ### KTransformers+SGLang Inference Deployment
43 Launch with KTransformers + SGLang for CPU+GPU heterogeneous inference:
44
45 ```
46 python -m sglang.launch_server \
47 --model path/to/Kimi-K2.5/ \
48 --kt-amx-weight-path path/to/Kimi-K2.5/ \
49 --kt-cpuinfer 64 \
50 --kt-threadpool-count 2 \
51 --kt-num-gpu-experts 180 \
52 --kt-amx-method AMXINT4 \
53 --trust-remote-code \
54 --mem-fraction-static 0.98 \
55 --chunked-prefill-size 16384 \
56 --max-running-requests 48 \
57 --max-total-tokens 50000 \
58 --tensor-parallel-size 8 \
59 --enable-p2p-check \
60 --disable-shared-experts-fusion
61 ```
62
63 Achieves 640.12 tokens/s Prefill and 24.51 tokens/s Decode (48-way concurrency) on 8× NVIDIA L20 + 2× Intel 6454S.
64
65 More details: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Kimi-K2.5.md .
66
67 ### KTransformers+LLaMA-Factory Fine-tuning Deployment
68
69 You can use below command to run LoRA SFT with KT+llamafactory.
70
71 ```
72 # For LoRA SFT
73 USE_KT=1 llamafactory-cli train examples/train_lora/kimik2_lora_sft_kt.yaml
74 # For Chat with model after LoRA SFT
75 llamafactory-cli chat examples/inference/kimik2_lora_sft_kt.yaml
76 # For API with model after LoRA SFT
77 llamafactory-cli api examples/inference/kimik2_lora_sft_kt.yaml
78 ```
79
80 This achieves end-to-end LoRA SFT Throughput: 44.55 token/s on 2× NVIDIA 4090 + Intel 8488C with 1.97T RAM and 200G swap memory.
81
82 More details refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/SFT_Installation_Guide_KimiK2.5.md .
83