Ternarization_instructions_27b_en.md
5.5 KB · 103 lines · markdown Raw
1 # Instructions for Ternarization and Export to GGUF/TQ2_0 — Qwen3.8-27B
2
3 Model: `/mnt/nfs_share/Qwen3_8/`, class `JiRackDeltaNet_27b.py`
4 (3:1 hybrid Gated-DeltaNet / Gated-Attention, `model_type: qwen3_5`).
5
6 ## Status (updated 2026-08-28)
7
8 ✅ **Done:**
9 - HF → custom format conversion, full-pipeline testing
10 - **MTP block implemented and integrated** (see section 0 below)
11 - GGUF conversion successful (pt → safetensors → GGUF, bf16)
12 - Smoke test passed: model responds correctly ("Paris" for "What is the capital of France?")
13 - **Baseline perplexity: PPL = 5.4957 ± 0.12** (wikitext-100k, c=2048, bf16, lambda=0.0)
14 - File: `/mnt/nfs_share/Qwen3_8/wiki_100k.txt` (use this same file for comparison after ternarization)
15 - Log: `/mnt/nfs_share/Qwen3_8/ppl_baseline_bf16.log` (final line, 11 chunks)
16
17 ⏳ **Next steps:** QAT warmup lambda 0.0→1.0, ternarization materialization, TQ2_0 export, final perplexity
18
19 Current checkpoint: `/mnt/nfs_share/Qwen3_8/qwen38_27b_checkpoint_migrated.pt` (53.79GB, lambda=0.0)
20 - Contains MTP weights (transferred from the original HF checkpoint), but not yet ternarized. - It will be converted into a new `.pt` file with native MTP support using the `convert_qwen38_to_jirack.py` script (updated version).
21
22 Quantization mechanism (same as the 32B model): `BitLinear` calculates gamma as the **per-tensor absmean**;
23 weights map to exactly three values ​​(`-gamma, 0, +gamma`), ensuring a lossless TQ2_0 round-trip.
24 **Difference from the 32B model:** two architectural adjustments—see sections 0 and 1.
25
26 ---
27
28 ## 0. MTP Block (Medusa Token Prediction, layer 64)
29
30 **What it is:** The new llama.cpp architecture (version ≥ ca3d5a3e1, tag b10665) requires
31 an MTP layer for models like `qwen3_5`. MTP is a speculative decoder (draft layer)
32 designed to improve batch generation, similar to Medusa. Implemented in `JiRackDeltaNet_27b.py`. **What has been done so far:**
33 - The `MTPBlock` class has been implemented in the code (15 tensors: fc, norms).
34 - MTP weights have been ported from the original Qwen3.8-27B HF checkpoint.
35 - They have been integrated into the GGUF file during conversion (`qwen38_27b_jirack_baseline_bf16.gguf`).
36 - MTP is **not** included in the ternarization process (it remains in full precision, ~100MB of weights).
37
38 **Why we aren't ternarizing MTP:**
39 - The MTP layer is not essential for standard generation (it is used only when the speculative decoding flag is enabled).
40 - Ternarizing the 400 main layers is sufficient to achieve the target compression.
41 - Keeping MTP in full precision preserves the quality of the speculative decoder for users who choose to enable it.
42
43 **For QAT warmup and materialization:** simply skip the `mtp.*` layers when applying
44 the `BitLinear` patch—this is already handled by `apply_bitlinear_patch(skip_mtp=True)`.
45
46 ---
47
48 ## 1. What gets ternarized and what doesn't (THREE EXCEPTIONS)
49
50 By default (`include_gate_proj=False` in `apply_bitlinear_patch`),
51 the following 400 linear layers are patched:
52 - `linear_attn.in_proj_qkv`, `linear_attn.in_proj_z`, `linear_attn.out_proj`
53 - `self_attn.q_proj`, `k_proj`, `v_proj`, `o_proj`
54 - `mlp.gate_proj`, `up_proj`, `down_proj`
55
56 **The following are NOT patched and must remain in full precision:**
57 - `linear_attn.in_proj_b` and `linear_attn.in_proj_a` — these are the gate scalars
58 for DeltaNet (dt/A-gate projections); they control recurrence stability. This was a deliberate decision: the risk of destabilizing the recurrence formula itself
59 through coarse ternary quantization was deemed greater than the potential
60 savings.
61 - All normalization layers (`RMSNorm`, `RMSNormGated`), `conv1d` (depthwise,
62 4-tap), `dt_bias`, `A_log`.
63 - `embed_tokens`, `lm_head`.
64
65 **Implication for GGUF export:** when quantizing to GGUF, you must explicitly
66 specify the tensor type for these layers (using `--tensor-type` or
67 the equivalent in your version of `llama-quantize`) so that
68 `in_proj_b`/`in_proj_a`, conv1d, and all normalization layers remain in f16/f32
69 rather than being subjected to the general TQ2_0 setting during a full-file
70 quantization pass. Without explicit specification, some quantization scripts
71 apply a single type to all eligible tensors indiscriminately—which would
72 silently break the recurrence mechanism.
73
74 ## 2. Architectural risk — ALREADY VERIFIED ✅
75
76 Gated DeltaNet is a hybrid architecture that is new to llama.cpp. The risk lay in the
77 fact that its support is less mature than that of Qwen2-attention in 32B models. **Verified (2026-08-28):**
78 - GGUF conversion (`pt` → `safetensors` → `qwen38_27b_jirack_baseline_bf16.gguf`):
79 successful, 866 tensors, 54.6GB
80 - Smoke test (llama-cli with chat-template): model responds correctly
81 ("The capital of France is Paris" with reasoning layer)
82 - Baseline perplexity on wikitext-100k: **PPL = 5.4957 ± 0.12**
83 (11 chunks of 2048 tokens, bf16, lambda=0.0)
84
85 **Conclusion:** the architecture works correctly in llama.cpp. Any further
86 degradation during ternary/TQ2_0 quantization will stem from the quantization
87 process itself, not from implementation bugs.
88
89 **Note for the future:** if perplexity spikes abnormally (>>6.5) during TQ2_0,
90 this is not an architectural risk—it indicates an issue with QAT warmup
91 or incorrectly excluded layers.
92
93 ## 3. QAT training (when ternary quantization is required)
94
95 1. Load checkpoint:
96 ```python
97 model = JiRackQwen38ForCausalLM.load_checkpoint(
98 "/data/qwen38_27b_checkpoint_migrated.pt", dtype=torch.bfloat16)
99 ```
100 2. Gradually ramp up the warmup lambda from 0.0 to 1.0 (similar to the 32B model,
101 using `set_lambda` on all modules that support this method).
102 3. Save the trained checkpoint.
103