JiRack_README_en.md
4.7 KB · 239 lines · markdown Raw
1 # JiRack Qwen 3.8-27B with System Prompt
2
3 This is the **Qwen 3.8-27B** GGUF model featuring:
4 - ✅ **Built-in system prompt** (no need to pass it manually)
5 - ✅ **Ternary quantization** (for speed)
6 - ✅ **Multi-Token Prediction** (predicts multiple tokens simultaneously)
7 - ✅ **Q4_K_M quantization** (16.8 GB, runs on modern CPUs/GPUs)
8
9 ---
10
11 ## Quick Start
12
13 ### 1. Install llama.cpp
14
15 ```bash
16 git clone https://github.com/ggerganov/llama.cpp
17 cd llama.cpp
18 make
19 # Or with GPU support:
20 # make LLAMA_CUDA=1
21 ```
22
23 ### 2. Run the model
24
25 **Interactive chat:**
26
27 ```bash
28 ./llama-cli -m JiRackDeltaNet_27b_with_prompt.Q4_K_M.gguf -c 4096 -n 512
29 ```
30
31 Parameters:
32 - `-c 4096` — context window (4K tokens)
33 - `-n 512` — maximum new tokens to generate
34 - `-ngl 99` — if using an NVIDIA GPU
35
36 **Via API server:**
37
38 ```bash
39 ./llama-server -m JiRackDeltaNet_27b_with_prompt.Q4_K_M.gguf --port 8000
40 ```
41
42 Then (in a separate terminal):
43
44 ```bash
45 curl http://localhost:8000/v1/chat/completions \
46 -H "Content-Type: application/json" \
47 -d '{
48 "messages": [
49 {"role": "user", "content": "What is quantum computing?"}
50 ],
51 "temperature": 0.7,
52 "max_tokens": 256
53 }'
54 ```
55
56 ---
57
58 ## Built-in System Prompt Prompt
59
60 The model automatically uses:
61
62 > You are JiRack, an advanced AI assistant based on Qwen 3.8-27B...
63
64 If you want to **override the prompt**, use the `-p` flag:
65
66 ```bash
67 ./llama-cli -m model.gguf -p "You are a helpful coding expert..."
68 ```
69
70 ---
71
72 ## Hardware Requirements
73
74 ### Minimum:
75 - **CPU**: Intel i5/AMD Ryzen 5+ (4+ cores)
76 - **RAM**: 32 GB (for a smooth experience; 16 GB works but will be slow)
77 - **Storage**: 20 GB free space (the file itself is 16.8 GB)
78
79 ### Recommended:
80 - **GPU**: NVIDIA RTX 3060+ / RTX 4060 (8 GB VRAM) — provides 30-50x acceleration
81 - **RAM**: 64 GB
82 - **NVMe SSD**: for fast model loading
83
84 ### Approximate Speed:
85
86 | Hardware | Speed ​​|
87 |----------|-------|
88 | CPU (i5-12400) | 1-2 tokens/sec |
89 | GPU (RTX 4060, 8GB) | 15-20 tokens/sec |
90 | GPU (RTX 4090) | 60+ tokens/sec |
91
92 ---
93
94 ## Usage Examples
95
96 ### Task 1: Answer a question
97
98 ```bash
99 ./llama-cli -m model.gguf -p "Explain machine learning in simple terms" -n 256
100 ```
101
102 ### Task 2: Write code
103
104 ```bash
105 ./llama-cli -m model.gguf -p "Write a Python function to sort a list" -n 512
106 ```
107
108 ### Task 3: Translate text
109
110 ```bash
111 ./llama-cli -m model.gguf -p "Translate to Russian: Hello, how are you?" -n 128
112 ```
113
114 ### Task 4: Analysis
115
116 ```bash
117 ./llama-cli -m model.gguf -p "Analyze the pros and cons of remote work" -n 512
118 ```
119
120 ---
121
122 ## Generation Parameters
123
124 ```bash
125 ./llama-cli -m model.gguf \
126 -c 4096 # Context window
127 -n 512 # Max new tokens
128 -t 8 # Threads (use the number of CPU cores)
129 -ngl 99 # Layers on GPU (if available)
130 --temp 0.7 # Temperature (0.0 = deterministic, 1.0 = random)
131 --top_p 0.9 # Nucleus sampling
132 -i # Interactive mode (default)
133 -p "Prompt" # Initial system prompt
134 ```
135
136 ---
137
138 ## API via llama-server
139
140 Start the server:
141
142 ```bash
143 ./llama-server -m model.gguf --port 8000
144 ```
145
146 ### Chat Completions (OpenAI-compatible):
147
148 ```bash
149 curl http://localhost:8000/v1/chat/completions \
150 -H "Content-Type: application/json" \
151 -d '{
152 "model": "jirack",
153 "messages": [
154 {
155 "role": "system",
156 "content": "You are a helpful assistant."
157 },
158 {
159 "role": "user",
160 "content": "What is 2+2?"
161 }
162 ],
163 "temperature": 0.7,
164 "max_tokens": 100,
165 "stream": false
166 }'
167 ```
168
169 ### Completions:
170
171 ```bash
172 curl http://localhost:8000/v1/completions \
173 -H "Content-Type: application/json" \
174 -d '{
175 "prompt": "Once upon a time",
176 "max_tokens": 256,
177 "temperature": 0.9
178 }'
179 ```
180
181 ---
182
183 ## Troubleshooting
184
185 ### Model is slow
186
187 1. Reduce context size: `-c 2048` instead of `-c 4096`
188 2. Reduce generation length: `-n 128` instead of `-n 512`
189 3. Use a GPU (provides 10–50x speedup)
190
191 ### Insufficient memory
192
193 1. Set `mlock=false`: `--no-mlock`
194 2. Reduce context size
195 3. Use lower quantization (e.g., Q3_K_M, if available)
196
197 ### CUDA errors
198
199 Ensure llama.cpp is compiled with GPU support:
200
201 ```bash
202 make clean
203 make LLAMA_CUDA=1
204 ```
205
206 ### Incorrect answers
207
208 Try adjusting the temperature:
209
210 ```bash
211 ./llama-cli -m model.gguf --temp 0.5 -p "Prompt" # More conservative
212 ./llama-cli -m model.gguf --temp 1.5 -p "Prompt" # More creative
213 ```
214
215 ---
216
217 ## Additional Information
218
219 - **Model**: Qwen 3.8-27B (127B parameters, compressed to 27B via ternary quantization)
220 - **Quantization**: Q4_K_M (balanced trade-off between quality and size)
221 - **Context**: Up to 4096 tokens (main window)
222 - **Language**: Multilingual (English, 中文, Russian, etc.)
223 - **MTP**: Multi-Token Prediction for accelerating speculative decoding
224
225 ---
226
227 ## License
228
229 The model is based on Qwen 3.8-27B (Qwen License Agreement).
230 Quantization and optimization: JiRack Project.
231
232 ---
233
234 ## Questions and Support
235
236 If you encounter issues:
237 1. Check the hardware requirements
238 2. Ensure that llama
239