README.md
8.8 KB · 239 lines · markdown Raw
1 ---
2 language:
3 - en
4 - zh
5 - ja
6 - ko
7 - fr
8 - es
9 - pt
10 - de
11 - it
12 - ru
13 - ar
14 - vi
15 - th
16 tags:
17 - text-generation
18 - ternary
19 - bitnet
20 - 1.58bit
21 - cpu
22 - gguf
23 - qwen2.5
24 - deepseek
25 - efficient
26 - low-memory
27 - jirack
28 - web-ui
29 - routing
30 - tool-call
31 - robotics
32 license: mit
33 ---
34 # JiRack Ultra 1B (CPU)
35 A fast and efficient ~1.5B model optimized for CPU inference. The model was refactored with BitNet features and an updated tokenizer that includes new **Routing**, **Tool call**, and **Robotics** tags. Built on a redesigned DeepSeek R1 architecture with native ternary (BitNet-style) support and ready-to-run GGUF quantizations.
36 - JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
37
38
39 # JiRack Ternary Architedure & JiRack Tokenizer
40 - Benefits high quality CPU inference TQ_2 on Llama.cpp and Ollama via QAT
41 - Robotcs, Routing, Coding, Multimedia, Advanced tool calling via CMSManhattan/JiRackPrecisionTokenizer
42
43 ## PARTNERSHIP
44
45 - NVIDIA
46 - FISERV
47
48
49 # Ollama production support
50 - We are working to support JiRack on Ollama for production systems also
51 - added Jirack chat without reasoning feature https://ollama.com/cmsmanhattan
52 - Follow fresh Ollama platform updates
53
54 # JiRack sevice options
55 - Current quantizations were done from the FP16 model, but the model allows for more compression thanks to its ternary architecture.
56 - If you need to do ternary compression, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
57 - Plus double QAT via ONNX QAT.
58 - Adapt train process to avoid catastrophic forgetting with NDA
59 - Adapt train process to avoid fast plato in training with NDA
60 - Convert model to TQ2_0 with support AVX2 and AVX-512 CPU instructions for high performance on CPU
61 - QAT for TQ_2 Llama.cpp Ternarization docs https://huggingface.co/CMSManhattan/JiRackUltra_1b/blob/main/QAT_to_Llama.cpp_GGUF_TQ2_0_JirackUltra_1b.md
62 - Adapts to agentic or instruct models for tool calling, using the JiRak tokenizer to enable high-quality tool calling on small models — built as a domain-specific tool expert.
63 - Deployment and scale
64
65
66 # JiRack Codding Agent IDE
67 - It is Agent Coding IDE for JiRack Models to run via Ollama on home PC
68 - It good choose for Agent Coding IDE such as Cursor , Windsurf IDE or Devin IDE etc but more safe that ask you to apply changes and review.
69 - Web site https://www.jirack.com
70 - Final release version https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip
71
72
73
74
75 # Spring Boot AI tool calls examples for JiRack Ultra series
76 - Tool call library on java for Enterprise https://github.com/alibaba/spring-ai-alibaba
77
78 # GoEx AI tool calls examples for JiRack Ultra series
79 - Tool call library on python https://github.com/ShishirPatil/gorilla
80
81 # JiRack Ultra 1 tool calls to boost tool call quality
82 - Use JiRack Precision tokenzer tags for tool calls with ToolBench https://github.com/OpenBMB/ToolBench
83 - https://huggingface.co/xalss/Qwen2-7B-Instruct-glaive-function-calling
84 - https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1
85 - Add JiRack tool call tags in the dataset and modify tool call processor if needed
86
87
88
89
90 # JiRack RoboTech
91 - Advanced Tokenizer with Robotics & Routing & Tool calls Tokenizer and other
92 - [CMSManhattan/JiRackPrecisionTokenizer](https://huggingface.co/CMSManhattan/JiRackPrecisionTokenizer)
93
94
95
96 ## Available Variants
97 | Tag | Quant | Size | Approx. RAM | Description |
98 |-----|-------|------|-------------|-------------|
99 | `cmsmanhattan/jirack-ultra-1b-cpu:latest` | Full | 0.55 GB | ~1.8 GB | Full ternary reference |
100 | `cmsmanhattan/jirack-ultra-1b-cpu-q4:latest` | Q4_K_M | 0.38 GB | ~1.4 GB | Recommended balance |
101 | `cmsmanhattan/jirack-ultra-1b-cpu-q3:latest` | Q3_K_M | 0.31 GB | ~1.2 GB | Good quality / size trade-off |
102 | `cmsmanhattan/jirack-ultra-1b-cpu-q2:latest` | Q2_K | 0.24 GB | ~1.0 GB | Maximum compression |
103 ## Quick Start
104 ### Run with Docker
105 **Default CPU (Q4 recommended)**
106 ```bash
107 docker run -d \
108 --name jirack_ultra_1b \
109 -p 7869:7869 \
110 --cpus=16 \
111 -e THREADS=16 \
112 -e THREADS_BATCH=16 \
113 --restart unless-stopped \
114 cmsmanhattan/jirack-ultra-1b-cpu-q4:latest
115 ```
116 **Q3**
117 ```bash
118 docker run -d \
119 --name jirack_ultra_1b \
120 -p 7869:7869 \
121 --cpus=16 \
122 -e THREADS=16 \
123 -e THREADS_BATCH=16 \
124 --restart unless-stopped \
125 cmsmanhattan/jirack-ultra-1b-cpu-q3:latest
126 ```
127 **Q2 (lowest memory)**
128 ```bash
129 docker run -d \
130 --name jirack_ultra_1b \
131 -p 7869:7869 \
132 --cpus=16 \
133 -e THREADS=16 \
134 -e THREADS_BATCH=16 \
135 --restart unless-stopped \
136 cmsmanhattan/jirack-ultra-1b-cpu-q2:latest
137 ```
138 **Full precision**
139 ```bash
140 docker run -d \
141 --name jirack_ultra_1b \
142 -p 7869:7869 \
143 --cpus=16 \
144 -e THREADS=16 \
145 -e THREADS_BATCH=16 \
146 --restart unless-stopped \
147 cmsmanhattan/jirack-ultra-1b-cpu:latest
148 ```
149 **Multi CPU**
150 ```bash
151 docker run -d \
152 --name jirack_ultra_1b \
153 -p 7869:7869 \
154 --cpus=16 \
155 -e THREADS=16 \
156 -e THREADS_BATCH=16 \
157 --restart unless-stopped \
158 --memory=4g \
159 --cpus=4 \
160 cmsmanhattan/jirack-ultra-1b-cpu-q4:latest
161 ```
162 ### Docker Compose Example
163 ```yaml
164 services:
165 jirack:
166 image: cmsmanhattan/jirack-ultra-1b-cpu-q4:latest
167 container_name: jirack_ultra_1b
168 ports:
169 - "7869:7869"
170 volumes:
171 - .:/app
172 - ./web:/app/web
173 environment:
174 - MAX_TOKENS=2048
175 - TEMPERATURE=0.7
176 - TOP_P=0.9
177 - DEFAULT_STREAM=False
178 - INTRA_THREADS=4
179 - USE_ENV_ALLOCATOR=1
180 - THREADS=16
181 - THREADS_BATCH=16
182 deploy:
183 resources:
184 limits:
185 memory: 4g
186 ```
187 ## Access the UI
188 Once the container is running, open your browser and navigate to:
189 `http://localhost:7869`
190 This opens the JiRack UI — a clean web interface.
191 ## Changing the Port
192 The listening port can be easily modified directly from the **Settings** panel within the JiRack UI.
193
194 ## Licensing
195
196 - Model weights are released under the MIT License — free to use, modify, and distribute for any purpose, including commercial. No royalties, no per-user fees, no subscription.
197 - The Docker image with UI and the pre-built Ollama quantizations are separate paid products. If you prefer to build your own secure deployment — take the weights, assemble your own stack, and you're done.
198 - The JiRack Ultra 1B model for Docker and Ollama is provided under a commercial license ($12 per user per year).
199 - All JiRack UI clients are provided under a commercial license.
200 - However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
201 For commercial licensing, cluster deployment, or enterprise use of JiRack models, please contact us.
202 - **JiRack MS Windows 11 Desktop Client (with Ollama API):**
203 https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
204 - **Live email chat with the model:** support@cmsmanhattan.com
205 ## Hardware Recommendations
206 ### Recommended Hardware for JiRack Ultra 1B (single Docker container)
207 | Use Case | CPU | RAM | Recommended Quant | Expected Speed | Recommendation |
208 |-------------------|------------------------------|----------|-------------------|---------------------|----------------|
209 | Recommended | Ryzen 5 / Intel i5 | 4–8 GB | Q4_K_M | Excellent interactive | Best choice |
210 | High Performance | Ryzen 7 / Intel i7 | 8–16 GB | Full / Q4 | Excellent | Excellent |
211 | Low Memory | Modern 4+ core CPU | 2–4 GB | Q3_K_M or Q2_K | Usable | Acceptable |
212 | Edge / Minimal | Laptop / SBC CPU | 2 GB | Q2_K | Acceptable | Budget option |
213 ## Important Memory Notes
214 Even though the quantized 1B models are very small, we recommend the following for best experience:
215 - Q4_K_M: 2–4 GB system RAM minimum
216 - Q3_K_M / Q2_K: 1.5–3 GB system RAM
217 - Full precision: 3–4 GB+ system RAM recommended
218 Reasons for extra headroom:
219 - KV-cache consumption during generation
220 - Runtime overhead and temporary buffers
221 - System stability and avoiding out-of-memory errors
222 - Room for larger context windows
223 **Minimum recommended (Q4):** 2–3 GB system RAM
224 **Ideal:** 4–8 GB system RAM
225 I added the default model in full precision. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
226 ## Architecture Notes
227 - **Refactored with BitNet features**: Native BitLinear ternary path (b1.58-style) with λ-warmup STE
228 - **Updated tokenizer**: Extended with new special tags for **Routing**, **Tool call**, and **Robotics**
229 - Base: Redesigned Llama-3.2-1B style (Hidden 2048, Intermediate 8192, 16 layers, GQA 32/8, vocab 128256)
230 - RoPE θ = 10000, RMSNorm ε = 1e-6
231 - Ready-to-run GGUF quantizations (Q2_K, Q3_K_M, Q4_K_M)
232 ## 📧 Contact & Licensing
233 For joint venture opportunities, hardware integration, or licensing inquiries:
234 - **Email:** grabko@cmsmanhattan.com
235 - **Phone:** +1 (516) 777-0945
236 - **Location:** New York, USA
237
238 ## License
239 MIT License