README.md
8.9 KB · 230 lines · markdown Raw
1 ---
2 language:
3 - en
4 - zh
5 - ja
6 - ko
7 - fr
8 - es
9 - pt
10 - de
11 - it
12 - ru
13 - ar
14 - vi
15 - th
16 tags:
17 - text-generation
18 - ternary
19 - bitnet
20 - 1.58bit
21 - cpu
22 - gguf
23 - qwen2.5
24 - deepseek
25 - efficient
26 - low-memory
27 - jirack
28 - web-ui
29 - routing
30 - tool-call
31 - robotics
32 license: mit
33 ---
34 # JiRack Ultra 14B (CPU)
35 A fast and efficient 14B model optimized for CPU inference. The model was refactored with BitNet features and an updated tokenizer that includes new **Routing**, **Media**, **Vision**, **Sound**, **Tool call**, and **Robotics** tags. Built on a DeepSeek R1-14B architecture with native ternary (BitNet-style) support and ready-to-run GGUF quantizations.
36 - JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
37
38
39
40 # JiRack Ternary Architedure & JiRack Tokenizer
41 - Benefits high quality CPU inference TQ_2 on Llama.cpp and Ollama via QAT
42 - Robotcs, Routing, Coding, Multimedia, Advanced tool calling via CMSManhattan/JiRackPrecisionTokenizer
43 -
44
45 # JiRack sevice options
46 - Current quantizations were done from the FP16 model, but the model allows for more compression thanks to its ternary architecture.
47 - If you need to do ternary compression, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
48 - Plus double QAT via ONNX QAT.
49 - Adapt train process to avoid catastrophic forgetting with NDA
50 - Adapt train process to avoid fast plato in training with NDA
51 - Convert model to TQ2_0 with support AVX2 and AVX-512 CPU instructions for high performance on CPU
52 - QAT for TQ_2 Llama.cpp Ternarization docs https://huggingface.co/CMSManhattan/JiRackUltra_14b/blob/main/QAT_to_Llama.cpp_GGUF_TQ2_0_JirackUltra_14b.md
53 - Adapts to agentic or instruct models for tool calling, using the JiRak tokenizer to enable high-quality tool calling on small models — built as a domain-specific tool expert.
54 - Deployment and scale
55
56 # JiRack Codding Agent IDE
57 - It is Agent Coding IDE for JiRack Models to run via Ollama on home PC
58 - It good choose for Agent Coding IDE such as Cursor , Windsurf IDE or Devin IDE etc but more safe that ask you to apply changes and review.
59 - Web site https://www.jirack.com
60 - Final release version https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip
61
62 # Ollama production support
63 - We are working to support JiRack on Ollama for production systems also
64 - added Jirack chat without reasoning feature https://ollama.com/cmsmanhattan
65 - Follow fresh Ollama platform updates
66
67
68 # Spring Boot AI tool calls examples for JiRack Ultra series
69 - Tool call library on java for Enterprise https://github.com/alibaba/spring-ai-alibaba
70
71 # GoEx AI tool calls examples for JiRack Ultra series
72 - Tool call library on python https://github.com/ShishirPatil/gorilla
73
74 # JiRack Ultra 1 tool calls to boost tool call quality
75 - Use JiRack Precision tokenzer tags for tool calls with ToolBench https://github.com/OpenBMB/ToolBench
76 - https://huggingface.co/xalss/Qwen2-7B-Instruct-glaive-function-calling
77 - https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1
78 - Add JiRack tool call tags in the dataset and modify tool call processor if needed
79
80
81
82 # JiRack RoboTech
83 - Advanced Tokenizer with Robotics & Routing & Tool calls Tokenizer and other
84 - [CMSManhattan/JiRackPrecisionTokenizer](https://huggingface.co/CMSManhattan/JiRackPrecisionTokenizer)
85
86 ## Available Variants
87 | Tag | Quant | Size | Approx. RAM | Description |
88 |-----|-------|------|-------------|-------------|
89 | `cmsmanhattan/jirack-ultra-14b-cpu:latest` | Full | 28.1 GB | ~28–32 GB | Full precision reference |
90 | `cmsmanhattan/jirack-ultra-14b-cpu-q4:latest` | Q4_K_M | 10.1 GB | ~8–12 GB | Recommended balance |
91 | `cmsmanhattan/jirack-ultra-14b-cpu-q3:latest` | Q3_K_M | 8.42 GB | ~7–10 GB | Good quality / size trade-off |
92 | `cmsmanhattan/jirack-ultra-14b-cpu-q2:latest` | Q2_K | 6.81 GB | ~6–9 GB | Maximum compression |
93 ## Quick Start
94 ### Run with Docker
95 - 14 B docker can be provided by request .
96 - Build docker on local from source or request fro me
97
98 **Default CPU (Q4 recommended)**
99 ```bash
100 docker run -d \
101 --name jirack_ultra_14b \
102 -p 7869:7869 \
103 --restart unless-stopped \
104 cmsmanhattan/jirack-ultra-14b-cpu-q4:latest
105 ```
106 **Q3**
107 ```bash
108 docker run -d \
109 --name jirack_ultra_14b \
110 -p 7869:7869 \
111 --cpus=16 \
112 -e THREADS=16 \
113 -e THREADS_BATCH=16 \
114 --restart unless-stopped \
115 cmsmanhattan/jirack-ultra-14b-cpu-q3:latest
116 ```
117 **Q2 (lowest memory)**
118 ```bash
119 docker run -d \
120 --name jirack_ultra_14b \
121 -p 7869:7869 \
122 --cpus=16 \
123 -e THREADS=16 \
124 -e THREADS_BATCH=16 \
125 --restart unless-stopped \
126 cmsmanhattan/jirack-ultra-14b-cpu-q2:latest
127 ```
128 **Full precision**
129 ```bash
130 docker run -d \
131 --name jirack_ultra_14b \
132 -p 7869:7869 \
133 --cpus=16 \
134 -e THREADS=16 \
135 -e THREADS_BATCH=16 \
136 --restart unless-stopped \
137 cmsmanhattan/jirack-ultra-14b-cpu:latest
138 ```
139 **Multi CPU**
140 ```bash
141 docker run -d \
142 --name jirack_ultra_14b \
143 -p 7869:7869 \
144 --cpus=16 \
145 -e THREADS=16 \
146 -e THREADS_BATCH=16 \
147 --restart unless-stopped \
148 --memory=16g \
149 --cpus=8 \
150 cmsmanhattan/jirack-ultra-14b-cpu-q4:latest
151 ```
152 ### Docker Compose Example
153 ```yaml
154 services:
155 jirack:
156 image: cmsmanhattan/jirack-ultra-14b-cpu-q4:latest
157 container_name: jirack_ultra_14b
158 ports:
159 - "7869:7869"
160 volumes:
161 - .:/app
162 - ./web:/app/web
163 environment:
164 - MAX_TOKENS=2048
165 - TEMPERATURE=0.7
166 - TOP_P=0.9
167 - DEFAULT_STREAM=False
168 - INTRA_THREADS=4
169 - USE_ENV_ALLOCATOR=1
170 - THREADS=16
171 - THREADS_BATCH=16
172 deploy:
173 resources:
174 limits:
175 memory: 16g
176 ```
177 ## Access the UI
178 Once the container is running, open your browser and navigate to:
179 `http://localhost:7869`
180 This opens the JiRack UI — a clean web interface.
181 ## Changing the Port
182 The listening port can be easily modified directly from the **Settings** panel within the JiRack UI.
183 ## Licensing
184 - Model weights are released under the MIT License — free to use, modify, and distribute for any purpose, including commercial. No royalties, no per-user fees, no subscription.
185 - The Docker image with UI and the pre-built Ollama quantizations are separate paid products. If you prefer to build your own secure deployment — take the weights, assemble your own stack, and you're done.
186 - The JiRack Ultra 14B model for Docker and Ollama is provided under a commercial license ($12 per user per year).
187 - All JiRack UI clients are provided under a commercial license.
188 - However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
189 For commercial licensing, cluster deployment, or enterprise use of JiRack models, please contact us.
190 - **JiRack MS Windows 11 Desktop Client (with Ollama API):**
191 https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
192 - **Live email chat with the model:** support@cmsmanhattan.com
193 ## Hardware Recommendations
194 ### Recommended Hardware for JiRack Ultra 14B (single Docker container)
195 | Use Case | CPU | RAM | Recommended Quant | Expected Speed | Recommendation |
196 |-------------------|------------------------------|----------|-------------------|---------------------|----------------|
197 | Recommended | Ryzen 7 / Intel i7 | 24–32 GB | Q4_K_M | Good interactive | Best choice |
198 | High Performance | Ryzen 9 / Intel i9 | 32–48 GB | Full / Q4 | Excellent | Excellent |
199 | Low Memory | Modern 8+ core CPU | 16–24 GB | Q3_K_M or Q2_K | Usable | Acceptable |
200 | Edge / Minimal | Strong laptop CPU | 16 GB | Q2_K | Acceptable | Budget option |
201 ## Important Memory Notes
202 Even though the quantized 14B models are relatively compact, we recommend the following for best experience:
203 - Q4_K_M: 16–24 GB system RAM minimum
204 - Q3_K_M / Q2_K: 12–20 GB system RAM
205 - Full precision: 32 GB+ system RAM recommended
206 Reasons for extra headroom:
207 - KV-cache consumption during generation
208 - Runtime overhead and temporary buffers
209 - System stability and avoiding out-of-memory errors
210 - Room for larger context windows
211 **Minimum recommended (Q4):** 16 GB system RAM
212 **Ideal:** 24–32 GB system RAM
213 I added the default model in full precision. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
214 ## Architecture Notes
215 - **Refactored with BitNet features**: Native BitLinear ternary path (b1.58-style) with λ-warmup STE
216 - **Updated tokenizer**: Extended with new special tags for **Routing**, **Tool call**, and **Robotics**
217 - Base: Qwen2.5-14B style (Hidden 5120, 48 layers, GQA 40/8, vocab 152064)
218 - RoPE θ = 1 000 000, RMSNorm ε = 1e-6
219 - Intermediate size: 13824
220 - Ready-to-run GGUF quantizations (Q2_K, Q3_K_M, Q4_K_M)
221 ## 📧 Contact & Licensing
222 For joint venture opportunities, hardware integration, or licensing inquiries:
223 - **Email:** grabko@cmsmanhattan.com
224 - **Phone:** +1 (516) 777-0945
225 - **Location:** New York, USA
226
227
228
229 ## License
230 MIT License