README.md
| 1 | --- |
| 2 | license: apache-2.0 |
| 3 | license_link: https://ai.google.dev/gemma/docs/gemma_4_license |
| 4 | language: |
| 5 | - en |
| 6 | base_model: |
| 7 | - google/gemma-4-26B-A4B-it |
| 8 | base_model_relation: quantized |
| 9 | tags: |
| 10 | - telecom |
| 11 | - telecommunications |
| 12 | - open-telco-ai |
| 13 | - classification |
| 14 | - typed-decisions |
| 15 | - decision-engine |
| 16 | - system-one |
| 17 | - gemma4 |
| 18 | - mixture-of-experts |
| 19 | - awq |
| 20 | - w4a16 |
| 21 | - compressed-tensors |
| 22 | - 4-bit |
| 23 | - amd |
| 24 | - rocm |
| 25 | - nvidia |
| 26 | - cuda |
| 27 | - h100 |
| 28 | - vllm |
| 29 | - gguf |
| 30 | - llama.cpp |
| 31 | pipeline_tag: text-generation |
| 32 | --- |
| 33 | |
| 34 | # OTel-2.0-Decision-26B-A4B-4bit |
| 35 | |
| 36 | > **Versioning notice:** This card describes the current OTel-2.0-Decision-26B-A4B-4bit checkpoint. The model weights may be updated over time. For reproducible evaluation or production deployment, pin a specific model revision, checkpoint hash, or release tag rather than relying on the floating latest version. |
| 37 | |
| 38 | Details on the training data, training method and configuration, data-processing pipeline, and training infrastructure will be added soon. |
| 39 | |
| 40 | **OTel-2.0-Decision-26B-A4B-4bit** is a 4-bit W4A16 build of Google's Gemma 4 26B-A4B-it for typed decisions: classification, Yes/No questions, and ordered levels. Questions are supplied at request time, and decisions are read from answer-label probabilities rather than generated prose. |
| 41 | |
| 42 | ## Intended Use |
| 43 | |
| 44 | Classify messages, call transcripts, tickets, and documents, or ask multiple questions about the same text. Use label probabilities for confidence thresholds and routing. This release is not intended for open-ended chat, retrieval, or reranking. |
| 45 | |
| 46 | ## Evaluation |
| 47 | |
| 48 | Benchmarks use the typed-decision readout (`level=L0`, no prior, shared-prefix execution, adaptive Yes/No ordering with margin 4.0). All runs reported zero errors and `answer_mass` of 1.00. |
| 49 | |
| 50 | | Benchmark | AMD MI350P | AMD Ryzen AI MAX+ PRO 395 (Strix Halo) | |
| 51 | |---|---:|---:| |
| 52 | | JevBench public (231 tasks) | **86.58% to 87.45%** (200 to 202 of 231) | **87.45%** (202/231) | |
| 53 | |
| 54 | JevBench by tier: 100% on easy, 94.44% to 95.83% on original, and 74.77% to 76.58% on hard tasks. The MoE kernel is not bit-deterministic, so results vary on a few low-margin tasks across runs. The Strix Halo run used vLLM 0.26.0 on ROCm 7.13 (48/48 easy, 71/72 original, 83/111 hard). |
| 55 | |
| 56 | Use a system message that asks for a single label (see *Prompting and Label Probabilities*). Without one, the model sometimes answers hard tasks in prose instead of a label, which lowers hard-tier accuracy by about 6 points. |
| 57 | |
| 58 | ## Inference Recipe |
| 59 | |
| 60 | This repository holds the same model in two formats. Pick the recipe that matches your engine: |
| 61 | |
| 62 | | Weights | Engine | API | Section | |
| 63 | |---|---|---|---| |
| 64 | | Safetensors, W4A16 (repository root) | vLLM 0.26 or 0.30 on ROCm or CUDA | `/v1/chat/completions` with logprobs | Options 1 and 2, *NVIDIA H100*, *Strix Halo* | |
| 65 | | Safetensors, W4A16 (repository root) | vLLM nightly | `/v1/systemone` | *System One API* | |
| 66 | | GGUF Q8_0 or Q4_K_M (`gguf/`) | llama.cpp `llama-server` (Vulkan, CUDA and other backends) | `/v1/chat/completions` with logprobs | *Inference with llama.cpp* | |
| 67 | | GGUF Q8_0_systemone or Q4_K_M_systemone (`gguf/`) | llama.cpp `llama-server` | `/v1/systemone` | *`/v1/systemone` with llama.cpp* | |
| 68 | |
| 69 | Serve on one AMD MI350P with vLLM 0.30.0. Both options use the same weights and API. For NVIDIA GPUs, see *NVIDIA H100: vLLM on CUDA* below. |
| 70 | |
| 71 | | Option | Image | Throughput | |
| 72 | |---|---|---:| |
| 73 | | 1. Official vLLM | `vllm/vllm-openai-rocm:v0.30.0` | 22.7k tok/s | |
| 74 | | 2. Prebuilt image | `farbodatdocker/otel-decision-vllm-rocm:v0.30.0` | 32.3k tok/s (42% more) | |
| 75 | |
| 76 | Throughput figures are peak aggregate output tokens per second. The highest throughput is reached only when the GPU runs near maximum utilization, with many concurrent requests in flight. A single request runs much slower, so batch or parallelize requests to reach these numbers. |
| 77 | |
| 78 | ### Download |
| 79 | |
| 80 | ```bash |
| 81 | pip install -U huggingface_hub |
| 82 | hf download farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit \ |
| 83 | --exclude "gguf/*" --local-dir ./OTel-2.0-Decision-26B-A4B-4bit |
| 84 | ``` |
| 85 | |
| 86 | `--exclude "gguf/*"` skips the four GGUF files (87.3 GB), which only `llama-server` needs. |
| 87 | |
| 88 | ### Option 1: Official vLLM |
| 89 | |
| 90 | ```bash |
| 91 | docker run --rm --name otel_decision \ |
| 92 | --device /dev/kfd --device /dev/dri --group-add video \ |
| 93 | --ipc=host --security-opt seccomp=unconfined -p 8000:8000 \ |
| 94 | -v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \ |
| 95 | vllm/vllm-openai-rocm:v0.30.0 \ |
| 96 | /model \ |
| 97 | --served-model-name OTel-2.0-Decision-26B-A4B-4bit \ |
| 98 | --max-model-len 16384 \ |
| 99 | --enable-prefix-caching \ |
| 100 | --max-logprobs 64 \ |
| 101 | --port 8000 |
| 102 | ``` |
| 103 | |
| 104 | With vLLM 0.30.0 already installed, the equivalent is: |
| 105 | |
| 106 | ```bash |
| 107 | vllm serve ./OTel-2.0-Decision-26B-A4B-4bit \ |
| 108 | --served-model-name OTel-2.0-Decision-26B-A4B-4bit \ |
| 109 | --max-model-len 16384 --enable-prefix-caching --max-logprobs 64 --port 8000 |
| 110 | ``` |
| 111 | |
| 112 | ### Option 2: Prebuilt Image |
| 113 | |
| 114 | The official image with a corrected Gemma 4 MoE kernel and MI350P tile configuration. |
| 115 | |
| 116 | ```bash |
| 117 | docker run --rm --name otel_decision \ |
| 118 | --device /dev/kfd --device /dev/dri --group-add video \ |
| 119 | --ipc=host --security-opt seccomp=unconfined -p 8000:8000 \ |
| 120 | -v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \ |
| 121 | farbodatdocker/otel-decision-vllm-rocm:v0.30.0 \ |
| 122 | /model \ |
| 123 | --served-model-name OTel-2.0-Decision-26B-A4B-4bit \ |
| 124 | --max-model-len 16384 \ |
| 125 | --enable-prefix-caching \ |
| 126 | --max-logprobs 64 \ |
| 127 | --moe-backend flydsl \ |
| 128 | --port 8000 |
| 129 | ``` |
| 130 | |
| 131 | Use `--moe-backend flydsl` only with Option 2. The first start takes 4 to 10 minutes. |
| 132 | |
| 133 | ### System One API (`/v1/systemone`) |
| 134 | |
| 135 | vLLM serves this model on its System One endpoint with no conversion. One request carries a state and any number of named questions (choice, Yes/No, score), and the reply gives a probability for every option. The endpoint is on vLLM main and in nightly builds, not yet in a release. Tested on one NVIDIA H100 80GB with `0.33.0rc1.dev613+g44965a109`, the vLLM main build of commit `44965a1`; the command below installs exactly that build. A newer nightly or release that includes `/v1/systemone` should also work: replace the pinned install line with `uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly`. |
| 136 | |
| 137 | ```bash |
| 138 | uv venv --python 3.12 --seed && source .venv/bin/activate |
| 139 | uv pip install "vllm==0.33.0rc1.dev613+g44965a109" --torch-backend=auto \ |
| 140 | --extra-index-url https://wheels.vllm.ai/44965a10974ee417f6f937eee207409462b2a0ab |
| 141 | |
| 142 | vllm serve farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit \ |
| 143 | --served-model-name OTel-2.0-Decision-26B-A4B-4bit \ |
| 144 | --max-model-len 16384 --enable-prefix-caching \ |
| 145 | --logprobs-mode raw_logprobs --port 8000 |
| 146 | ``` |
| 147 | |
| 148 | `--logprobs-mode raw_logprobs` is required. With prefix caching, the state is prefilled once and each question only adds its own tokens. |
| 149 | |
| 150 | ```bash |
| 151 | curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ |
| 152 | "model": "OTel-2.0-Decision-26B-A4B-4bit", |
| 153 | "instructions": "You are a decision function. You will be given a state and one question. Reply with the answer label only: no words, no punctuation, no explanation.", |
| 154 | "state": "Caller: There is a 14 dollar charge on my bill I do not recognise. I want it refunded. I already rebooted the router twice.", |
| 155 | "questions": { |
| 156 | "refund_requested": {"type": "noul", "instructions": "Did the caller explicitly ask for a refund or credit?"}, |
| 157 | "route": {"type": "choice", "instructions": "Which team should handle this call?", |
| 158 | "criteria": {"billing": "Charges, refunds and payments", "technical": "Internet, TV and equipment faults", "sales": "New plans and upgrades"}}, |
| 159 | "urgency": {"type": "score", "instructions": "How urgent is this call?", "criteria": ["Routine", "Soon", "Today"]} |
| 160 | } |
| 161 | }' |
| 162 | ``` |
| 163 | |
| 164 | Each answer has `probabilities` for every option and `confidence`; a Yes/No answer's `noul` is the probability of yes. `diagnostics.<id>.label_mass` is the probability the model put on the answer labels and should be close to 1.0. Pass the system prompt in `instructions` as above: without it, `label_mass` falls close to zero on some hard multiple choice questions. |
| 165 | |
| 166 | | Measured on `/v1/systemone` (one request at a time) | Accuracy | Latency mean / median / p95 | |
| 167 | |---|---:|---:| |
| 168 | | JevBench public, 231 tasks, one question per request | 85.28% (197/231) | 0.030 / 0.019 / 0.064 s | |
| 169 | | Synthetic call transcripts, 150 transcripts with 27 Yes/No questions each, one request per transcript | 99.80% | 0.114 / 0.112 / 0.148 s per transcript | |
| 170 | |
| 171 | JevBench is lower than in *Evaluation* above because the endpoint writes its own prompt and reads each question once, without the adaptive Yes/No ordering. |
| 172 | |
| 173 | ### Inference with llama.cpp (GGUF) |
| 174 | |
| 175 | The `gguf/` folder holds two GGUF files for `llama-server` chat completions, made from the BF16 weights with llama.cpp's own converter. Use llama.cpp release v0.9.0 or newer. |
| 176 | |
| 177 | | File | Size | When to use | |
| 178 | |---|---:|---| |
| 179 | | `OTel-2.0-Decision-26B-A4B-Q8_0.gguf` | 26.9 GB | Recommended: about the same accuracy as the safetensors | |
| 180 | | `OTel-2.0-Decision-26B-A4B-Q4_K_M.gguf` | 16.8 GB | Less memory, about 2.6 points lower on JevBench | |
| 181 | |
| 182 | ```bash |
| 183 | llama-server -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q8_0 \ |
| 184 | -ngl 999 -c 16384 --jinja --host 0.0.0.0 --port 8080 |
| 185 | ``` |
| 186 | |
| 187 | `-hf` downloads only the chosen GGUF, not the safetensors; use `:Q4_K_M` for the smaller file. Get `llama-server` for Vulkan from the llama.cpp releases page, or build it with `cmake -B build -DGGML_VULKAN=ON && cmake --build build -j` (needs the Vulkan SDK, or `libvulkan-dev` and `glslc`); the binary is `build/bin/llama-server`. To use a downloaded file, replace `-hf ...` with `-m OTel-2.0-Decision-26B-A4B-Q8_0.gguf`. |
| 188 | |
| 189 | Read the label probabilities from `/v1/chat/completions`, as in *Prompting and Label Probabilities* below, with `max_tokens` 1, `temperature` 0, `top_logprobs` 64 and `"chat_template_kwargs": {"enable_thinking": false}`. For `/v1/systemone`, use the `_systemone` files below. |
| 190 | |
| 191 | ```bash |
| 192 | curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{ |
| 193 | "messages": [ |
| 194 | {"role": "system", "content": "You are a decision function. You will be given a state and one question. Reply with the answer label only: no words, no punctuation, no explanation."}, |
| 195 | {"role": "user", "content": "State:\nCaller: There is a 14 dollar charge on my bill I do not recognise. I want it refunded.\n\nQuestion: Did the caller explicitly ask for a refund or credit?\nAnswer Yes or No."} |
| 196 | ], |
| 197 | "max_tokens": 1, "temperature": 0, "logprobs": true, "top_logprobs": 64, |
| 198 | "chat_template_kwargs": {"enable_thinking": false} |
| 199 | }' |
| 200 | ``` |
| 201 | |
| 202 | The reply's `top_logprobs` should put almost all probability on `Yes`. |
| 203 | |
| 204 | | `llama-server` on 1x H100 (CUDA), one request at a time | JevBench, 231 tasks (median per task) | Synthetic transcripts, 27 Yes/No questions each (mean per transcript) | |
| 205 | |---|---:|---:| |
| 206 | | Q8_0 | 87.01% (201/231), 0.072 s | 99.85%, 2.753 s | |
| 207 | | Q4_K_M | 84.42% (195/231), 0.072 s | 99.75%, 2.195 s | |
| 208 | |
| 209 | On 70 JevBench Yes/No tasks whose criteria are labelled `true`/`false`, the model answers `true` or `false`, so `Yes`/`No` carry little probability and the decision is read from those small values, as in the published evaluation. Each transcript is 27 separate requests here, which is why it is slower than on vLLM. |
| 210 | |
| 211 | On AMD Ryzen AI MAX+ PRO 395 (Radeon 8060S, Vulkan, llama.cpp v0.9.0), both files gave the expected answer to the request above on 20 of 20 runs, with a median of 0.077 s (Q8_0, 27.1 GiB of GPU memory) and 0.058 s (Q4_K_M, 17.6 GiB) per request after the first. Validate on your own labelled data. |
| 212 | |
| 213 | #### `/v1/systemone` with llama.cpp |
| 214 | |
| 215 | The `gguf/` folder also holds `OTel-2.0-Decision-26B-A4B-Q8_0_systemone.gguf` and `OTel-2.0-Decision-26B-A4B-Q4_K_M_systemone.gguf`: the same weights with metadata that turns on `llama-server`'s own `/v1/systemone`. They take the same request as the vLLM *System One API* above and answer every question of a request in one call. These files answer decisions only (chat returns 501), and their system prompt is built in, so a request's `instructions` field has no effect. |
| 216 | |
| 217 | ```bash |
| 218 | llama-server -hf farbodtavakkoli/OTel-2.0-Decision-26B-A4B-4bit:Q8_0_systemone \ |
| 219 | -ngl 999 -c 65536 -np 8 -fa on --swa-full --host 0.0.0.0 --port 8081 |
| 220 | ``` |
| 221 | |
| 222 | Use `:Q4_K_M_systemone` for the smaller file. `--swa-full` is required: without it the questions of one request do not share the state, and answers can change between identical requests. |
| 223 | |
| 224 | ```bash |
| 225 | curl -s localhost:8081/v1/systemone -H 'content-type: application/json' -d '{ |
| 226 | "state": "Caller: There is a 14 dollar charge on my bill I do not recognise. I want it refunded. I already rebooted the router twice.", |
| 227 | "questions": { |
| 228 | "refund_requested": {"type": "noul", "instructions": "Did the caller explicitly ask for a refund or credit?"}, |
| 229 | "route": {"type": "choice", "instructions": "Which team should handle this call?", |
| 230 | "criteria": {"billing": "Charges, refunds and payments", "technical": "Internet, TV and equipment faults", "sales": "New plans and upgrades"}}, |
| 231 | "urgency": {"type": "score", "instructions": "How urgent is this call?", "criteria": ["Routine", "Soon", "Today"]} |
| 232 | } |
| 233 | }' |
| 234 | ``` |
| 235 | |
| 236 | A Yes/No answer is `{"type": "noul", "noul": <probability of yes>}`; choice and score answers have `probabilities` for every option, as on vLLM. The reply has no `diagnostics` field. |
| 237 | |
| 238 | | `/v1/systemone` with `llama-server`, one request at a time | JevBench, 231 tasks (median per task) | Synthetic transcripts, 27 Yes/No questions each (mean per transcript) | |
| 239 | |---|---:|---:| |
| 240 | | Q8_0_systemone, 1x H100 (CUDA) | 87.88% (203/231), 0.031 s | 99.75%, 0.588 s | |
| 241 | | Q4_K_M_systemone, 1x H100 (CUDA) | 86.58% (200/231), 0.032 s | 99.78%, 0.589 s | |
| 242 | |
| 243 | On AMD Ryzen AI MAX+ PRO 395 (Radeon 8060S, Vulkan, llama.cpp v0.9.0), both files gave the expected answers to a five question request on 20 of 20 runs, with a median of 0.515 s (Q8_0, 39.7 GiB of GPU memory) and 0.472 s (Q4_K_M, 30.1 GiB) per request. |
| 244 | |
| 245 | ### NVIDIA H100: vLLM on CUDA |
| 246 | |
| 247 | Serve on one NVIDIA H100 80GB (SXM, HBM3) with the official CUDA image of vLLM 0.26.0. vLLM selects the Marlin W4A16 kernels for both the dense and MoE layers automatically, and no config shim or extra flags are needed. |
| 248 | |
| 249 | ```bash |
| 250 | docker run --rm --name otel_decision \ |
| 251 | --gpus '"device=0"' \ |
| 252 | --ipc=host -p 8000:8000 \ |
| 253 | -v "$PWD/OTel-2.0-Decision-26B-A4B-4bit:/model:ro" \ |
| 254 | vllm/vllm-openai:v0.26.0 \ |
| 255 | /model \ |
| 256 | --served-model-name OTel-2.0-Decision-26B-A4B-4bit \ |
| 257 | --max-model-len 16384 \ |
| 258 | --enable-prefix-caching \ |
| 259 | --max-logprobs 64 \ |
| 260 | --port 8000 |
| 261 | ``` |
| 262 | |
| 263 | With vLLM 0.26.0 already installed (`pip install vllm==0.26.0`), the equivalent is: |
| 264 | |
| 265 | ```bash |
| 266 | CUDA_VISIBLE_DEVICES=0 vllm serve ./OTel-2.0-Decision-26B-A4B-4bit \ |
| 267 | --served-model-name OTel-2.0-Decision-26B-A4B-4bit \ |
| 268 | --max-model-len 16384 --enable-prefix-caching --max-logprobs 64 --port 8000 |
| 269 | ``` |
| 270 | |
| 271 | | Measured (1x H100 80GB, vLLM 0.26.0, 512 output tokens per request) | Value | |
| 272 | |---|---:| |
| 273 | | Peak aggregate output, 256 concurrent requests | 10.6k tok/s | |
| 274 | | 1,024 concurrent requests | 10.1k tok/s | |
| 275 | | Single request | 223 tok/s | |
| 276 | |
| 277 | The weights use about 17 GB, which leaves about 51 GB for KV cache (about 242k tokens) at the default `--gpu-memory-utilization 0.92`. The first start takes about 4 minutes because of torch.compile and CUDA graph capture. Later starts reuse the compile cache. To serve several replicas on one multi-GPU node, add `--data-parallel-size N`. |
| 278 | |
| 279 | ### AMD Ryzen AI MAX+ PRO 395 (Strix Halo): vLLM on ROCm |
| 280 | |
| 281 | Serve on one AMD Ryzen AI MAX+ PRO 395 (Strix Halo: Radeon 8060S GPU, 128 GB unified memory, configured as 96 GB GPU and 32 GB NPU) with vLLM 0.26.0 and ROCm 7.13 for gfx1151. |
| 282 | |
| 283 | ```bash |
| 284 | # 1. ROCm 7.13 + vLLM for gfx1151 (managed by the ROCm CLI) |
| 285 | rocm engines install vllm |
| 286 | PY=~/.rocm/runtimes/wheel/release-wheel-gfx1151-7-13-0/bin/python |
| 287 | |
| 288 | # 2. Config shim: transformers 5.19 drops Gemma 4's global head sizes that vLLM 0.26 reads |
| 289 | mkdir -p shim && cat > shim/sitecustomize.py <<'EOF' |
| 290 | from transformers import PretrainedConfig |
| 291 | from transformers.models.gemma4 import configuration_gemma4 as g4 |
| 292 | PretrainedConfig.allow_global_per_layer_attribute_access = True |
| 293 | _init = g4.Gemma4TextConfig.__init__ |
| 294 | def init(self, *a, **k): |
| 295 | _init(self, *a, **k) |
| 296 | lt, pl = list(getattr(self, "layer_types", []) or []), getattr(self, "per_layer_config", None) |
| 297 | if pl is not None and "full_attention" in lt: |
| 298 | full = pl[lt.index("full_attention")] |
| 299 | for dst, src in (("global_head_dim", "head_dim"), ("num_global_key_value_heads", "num_key_value_heads")): |
| 300 | if dst not in self.__dict__ and getattr(full, src, None) is not None: |
| 301 | object.__setattr__(self, dst, getattr(full, src)) |
| 302 | g4.Gemma4TextConfig.__init__ = init |
| 303 | EOF |
| 304 | |
| 305 | # 3. Serve (the torch wheel links OpenMPI 4's libmpi_cxx.so.40; point at its lib dir) |
| 306 | LD_LIBRARY_PATH=/path/to/openmpi4/lib PYTHONPATH=$PWD/shim \ |
| 307 | $PY -m vllm.entrypoints.cli.main serve ./OTel-2.0-Decision-26B-A4B-4bit \ |
| 308 | --served-model-name OTel-2.0-Decision-26B-A4B-4bit \ |
| 309 | --max-model-len 16384 --enable-prefix-caching --max-logprobs 64 \ |
| 310 | --gpu-memory-utilization 0.55 --max-num-seqs 256 --port 8000 |
| 311 | ``` |
| 312 | |
| 313 | | Measured (AC power, performance profile) | Value | |
| 314 | |---|---:| |
| 315 | | Peak aggregate output, 256 concurrent requests | 377 tok/s | |
| 316 | | 64 concurrent requests | 257 tok/s | |
| 317 | | Single request | 30 tok/s | |
| 318 | | JevBench public, 8 concurrent | 1.27 tasks/s | |
| 319 | |
| 320 | Peak throughput again needs near-maximum utilization: aggregate output keeps rising up to 256 concurrent requests. Keep HIP graphs on (the default). `--enforce-eager` cuts single-request speed to about 19 tok/s. The shim is not needed with vLLM releases that read Gemma 4's per-layer config directly. torch fails to import without an OpenMPI 4.x runtime. Distributions that ship only OpenMPI 5 (for example Ubuntu 26.04) lack `libmpi_cxx.so.40`, so extract an OpenMPI 4 package into a local directory and use that directory; no `sudo` is needed. |
| 321 | |
| 322 | ### Prompting and Label Probabilities |
| 323 | |
| 324 | Put the text first, then the question and labels. Use letters (`A`, `B`, …) for options or Yes/No, and numbers for ordered levels. Add a system message that asks for exactly one label, and request only the label through the chat endpoint, which applies the repository's template with thinking disabled. |
| 325 | |
| 326 | ```python |
| 327 | import math |
| 328 | from openai import OpenAI |
| 329 | |
| 330 | client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed") |
| 331 | question = """State: |
| 332 | Order #1182. Status: shipped on 3 September. Carrier: DHL. |
| 333 | |
| 334 | Question: Has the order been shipped? Answer strictly from the facts stated in the text. |
| 335 | Options: |
| 336 | A. Yes |
| 337 | B. No |
| 338 | Answer with the letter only.""" |
| 339 | labels = ["A", "B"] |
| 340 | |
| 341 | r = client.chat.completions.create( |
| 342 | model="OTel-2.0-Decision-26B-A4B-4bit", |
| 343 | temperature=0, |
| 344 | max_tokens=1, |
| 345 | logprobs=True, |
| 346 | top_logprobs=64, |
| 347 | messages=[ |
| 348 | {"role": "system", "content": "You are a typed decision engine. Reply with exactly one answer label " |
| 349 | "(a single letter or number) and nothing else. Do not explain."}, |
| 350 | {"role": "user", "content": question}, |
| 351 | ], |
| 352 | ) |
| 353 | top = {t.token: t.logprob for t in r.choices[0].logprobs.content[0].top_logprobs} |
| 354 | probs = {label: math.exp(top[label]) if label in top else 0.0 for label in labels} |
| 355 | print("probabilities:", probs) |
| 356 | print("decision:", max(probs, key=probs.get)) |
| 357 | print("answer_mass:", sum(probs.values())) |
| 358 | # Expected: decision A, answer_mass approximately 1.0 |
| 359 | ``` |
| 360 | |
| 361 | Choose the label with the highest probability. `answer_mass` is the summed probability of the answer labels and should be close to 1.0; a low value indicates an invalid decision readout. For multiple questions, keep the shared text byte-identical so prefix caching can reuse it. |
| 362 | |
| 363 | ### Verify the Deployment |
| 364 | |
| 365 | Check that `http://localhost:8000/v1/models` lists the branded model name, then run the example above and confirm decision `A` with `answer_mass` near 1.0. For `llama-server`, use port 8080 and the request in the llama.cpp section instead; `/v1/models` lists the GGUF path given to `-m` (or a name set with `--alias`), and the answer should be `Yes`. Validate representative prompts against a reference run before deployment. |
| 366 | |
| 367 | ## Limitations and Responsible Use |
| 368 | |
| 369 | Evaluation covers English text only. JevBench is small and does not guarantee production accuracy. Validate on your own human-labeled data, and do not assume label probabilities are calibrated for your workload. Keep human review for consequential decisions. |
| 370 | |
| 371 | The recipe starts an unauthenticated server. Add authentication before exposing it beyond your machine. |
| 372 | |
| 373 | ## Resources |
| 374 | |
| 375 | - [Project Page](https://huggingface.co/farbodtavakkoli) |
| 376 | - [OTel code](https://github.com/farbodtavakkoli/OTel) |
| 377 | - [OTel Selected Coverage](https://github.com/farbodtavakkoli/OTel/blob/main/docs/OTel-selected-coverage.md) |
| 378 | - [OTel Paper](https://arxiv.org/pdf/2610.07766) |
| 379 | - [OTel 1.0 Coverage](https://github.com/farbodtavakkoli/OTel/blob/main/docs/OTel-1.0-coverage.md) |
| 380 | - [OTel 2.0 Coverage](https://github.com/farbodtavakkoli/OTel/blob/main/docs/OTel-2.0-coverage.md) |
| 381 | - [OTel Docker](https://hub.docker.com/u/farbodatdocker) |
| 382 | |
| 383 | ## Citation |
| 384 | |
| 385 | ```bibtex |
| 386 | @inproceedings{tavakkoli2026otel2, |
| 387 | title = {OTel 2.0: Open Telco AI Datasets, Benchmarks, and Models}, |
| 388 | author = {Tavakkoli, Farbod and others}, |
| 389 | booktitle = {TBD}, |
| 390 | year = {2026}, |
| 391 | url = {https://huggingface.co/farbodtavakkoli} |
| 392 | } |
| 393 | ``` |
| 394 | |
| 395 | ## Contact |
| 396 | |
| 397 | If you have any technical questions, please feel free to reach out to [mark.austin@att.com](mailto:mark.austin@att.com) or [farbod.tavakkoli@att.com](mailto:farbod.tavakkoli@att.com) or [farbodtavakoli@gmail.com](mailto:farbodtavakoli@gmail.com) |
| 398 | |