HOW_TO_EMBED_PROMPT_en.md
3.4 KB · 168 lines · markdown Raw
1 # Embedding a system prompt into GGUF
2
3 ## What are the benefits?
4
5 Instead of end-users having to pass the system prompt every time:
6
7 ```bash
8 ./llama-cli -m model.gguf -p "You are JiRack..."
9 ```
10
11 They can simply run:
12
13 ```bash
14 ./llama-cli -m model_with_prompt.gguf
15 ```
16
17 The prompt will already be embedded inside. ---
18
19 ## Step 1: Prepare the prompt
20
21 Create a text file containing the system prompt, for example `system_prompt.txt`:
22
23 ```
24 You are JiRack, an advanced AI assistant based on Qwen 3.8-27B...
25 ```
26
27 ---
28
29 ## Step 2: Run the script
30
31 On your machine (where the files are stored):
32
33 ```bash
34 python embed_prompt_gguf_proper.py \
35 /path/to/JiRackDeltaNet_27b.Q4_K_M.gguf \
36 /path/to/JiRackDeltaNet_27b_with_prompt.Q4_K_M.gguf \
37 system_prompt.txt
38 ```
39
40 **Example for your project:**
41
42 ```bash
43 cd /home/claude
44 python embed_prompt_gguf_proper.py \
45 /mnt/nfs_share/Qwen3_8/JiRackDeltaNet_27b.Q4_K_M.gguf \
46 /mnt/nfs_share/Qwen3_8/JiRackDeltaNet_27b_with_prompt.Q4_K_M.gguf \
47 jirack_system_prompt.txt
48 ```
49
50 ---
51
52 ## Step 3: Distribute to users
53
54 Two files will be ready:
55
56 1. **`JiRackDeltaNet_27b_with_prompt.Q4_K_M.gguf`** — the main model file with the embedded prompt
57 2. **`JiRackDeltaNet_27b_with_prompt_SYSTEM_PROMPT.md`** — documentation regarding the embedded prompt
58
59 Distribute both files to users.
60
61 ---
62
63 ## How does the user run the model?
64
65 ### Using the llama.cpp CLI:
66
67 ```bash
68 ./llama-cli -m JiRackDeltaNet_27b_with_prompt.Q4_K_M.gguf
69 ```
70
71 The prompt will be loaded automatically.
72
73 ### Using llama-server (for the API):
74
75 ```bash
76 ./llama-server -m JiRackDeltaNet_27b_with_prompt.Q4_K_M.gguf
77 ```
78
79 Then the request:
80
81 ```bash
82 curl http://localhost:8000/v1/chat/completions \
83 -H "Content-Type: application/json" \
84 -d '{
85 "messages": [
86 {"role": "user", "content": "Hello, who are you?"}
87 ]
88 }'
89 ```
90
91 ---
92
93 ## How does it work?
94
95 The script:
96 1. **Copies** the original GGUF file
97 2. **Creates** a sidecar documentation file containing the prompt information
98 3. Allows the user to override the prompt using the `-p` flag if needed
99
100 > **Note:** If you want to embed the prompt *directly into the GGUF metadata*, you would need a more complex approach using `gguf-py`. The current method involves adding sidecar documentation, which is also very convenient.
101
102 ---
103
104 ## What if I need a different prompt?
105
106 Simply create a new file and repeat the process:
107
108 ```bash
109 python embed_prompt_gguf_proper.py \
110 original.gguf \
111 version_english.gguf \
112 english_prompt.txt
113 ```
114
115 Or for Russian:
116
117 ```bash
118 python embed_prompt_gguf_proper.py \
119 original.gguf \
120 version_russian.gguf \
121 russian_prompt.txt
122 ```
123
124 ---
125
126 ## File size?
127
128 The size remains **exactly the same**—the prompt is only about 500 bytes of text, so it fits easily into the metadata.
129
130 Original: 16.8 GB
131 With prompt: 16.8 GB (virtually unchanged)
132
133 ---
134
135 ## Ready-made prompt examples
136
137 ### Minimal prompt:
138
139 ```
140 You are JiRack, an AI assistant.
141 Be helpful, accurate, and concise.
142 ``` ```
143
144 ### Full version (including thinking instructions):
145
146 ```
147 You are JiRack, an advanced AI assistant.
148
149 When asked complex questions:
150 - Think step by step
151 - Show your reasoning
152 - Use [Start thinking]...[End thinking] for internal monologue
153
154 Be concise but thorough. Admit uncertainty.
155 ```
156
157 ### For a specific task (e.g., coding):
158
159 ```
160 You are JiRack Code Assistant.
161 - Provide clean, well-commented code
162 - Use best practices and design patterns
163 - Explain your solution
164 - Test edge cases
165 ```
166
167 Choose the one that fits your needs 👍
168