README.md
| 1 | --- |
| 2 | language: |
| 3 | - en |
| 4 | library_name: transformers |
| 5 | pipeline_tag: text-generation |
| 6 | base_model: HuggingFaceTB/SmolLM3-3B |
| 7 | license: other |
| 8 | license_name: webai-non-commercial-license-ver.-1.0 |
| 9 | license_link: https://huggingface.co/webAI-Official/webAI-ColVec1-4b/blob/main/LICENSE.md |
| 10 | tags: |
| 11 | - formal-logic |
| 12 | - reasoning |
| 13 | - lora |
| 14 | - model-merging |
| 15 | - wise-ft |
| 16 | - reinforcement-learning |
| 17 | - grpo |
| 18 | - smollm3 |
| 19 | - twil-lm |
| 20 | --- |
| 21 | |
| 22 | # TwIL-LM3 |
| 23 | |
| 24 | A 3B reasoning model for **formal logic** tasks, built from |
| 25 | [`HuggingFaceTB/SmolLM3-3B`](https://huggingface.co/HuggingFaceTB/SmolLM3-3B) through LoRA |
| 26 | supervised fine-tuning, checkpoint fusion, WiSE-FT weight interpolation, and entropy-weighted |
| 27 | GRPO reinforcement learning. |
| 28 | |
| 29 | It improves in-domain formal-logic performance by **+26% relative** over its base model |
| 30 | (macro gate 0.336 → 0.422) **and improves held-out benchmark performance at the same time** |
| 31 | (+0.022 core average). It is the only arm in this project that gains on both tracks, which is |
| 32 | why it is the recommended release of the pair. |
| 33 | |
| 34 |  |
| 35 | |
| 36 | ## Highlights |
| 37 | |
| 38 | * **Gains on both tracks at once** — the only arm in this project that does. In-domain macro gate |
| 39 | 0.336 → 0.422, and the held-out 10-dataset macro 0.7193 → 0.7339 rather than the usual collapse |
| 40 | that follows task-specific fine-tuning. |
| 41 | * **Beats every arm up to and including LFM2.5-8B-A1B** — roughly three times its parameter count |
| 42 | — on all six Track A objective lanes and all four summary rows, not on average alone. |
| 43 | * **Competitive with 8B on strict scoring.** On strict-7, which gives no loose-match credit |
| 44 | anywhere, it sits 0.012 behind Qwen3-8B (0.1971 against 0.2093) at 2.6x fewer parameters, and |
| 45 | ahead of it on Lean formalisation (token-F1 0.5869 against 0.4022) and semantic parsing (0.4416 |
| 46 | against 0.4257). |
| 47 | * **Structured formal output.** Tuned for the objects rather than the prose: FOL translation, |
| 48 | entailment labels, semantic parses, Lean statements and Lean proof critique. |
| 49 | * **The most efficient arm measured, at any scale.** 482-token Track B generations and 32.9 |
| 50 | completed answers per second — about eight times gpt-oss-120b's rate — because it answers |
| 51 | short rather than because it decodes unusually fast. |
| 52 | * **Lowest maths-corpus perplexity of any released arm in the table** (3.8229), including |
| 53 | Qwen3-8B at 4.0083. |
| 54 | * **Runs anywhere.** 3.08B parameters in bf16, with Q4\_K\_M GGUF at 1.78 GiB for CPU or 4 GB of |
| 55 | VRAM. |
| 56 | |
| 57 | It is not a general assistant: there is no safety or preference tuning here beyond what SmolLM3 |
| 58 | carries, and instruction following regressed slightly. See |
| 59 | [Limitations](#limitations-and-caveats). |
| 60 | |
| 61 | ## Model Details |
| 62 | |
| 63 | | Property | Value | |
| 64 | | ------------------------- | ------------------------------------------------------------------------------------------- | |
| 65 | | Model ID | `webAI-Official/TwIL-LM3` | |
| 66 | | Base model | [`HuggingFaceTB/SmolLM3-3B`](https://huggingface.co/HuggingFaceTB/SmolLM3-3B) | |
| 67 | | Total parameters | 3.08B | |
| 68 | | Architecture | SmolLM3 decoder-only transformer; 36 layers, hidden size 2048 | |
| 69 | | Input / output | Text / text | |
| 70 | | Language | English | |
| 71 | | Tokenizer vocabulary size | 128,256 | |
| 72 | | Context window | 65,536 tokens | |
| 73 | | Checkpoint precision | bfloat16 (5.73 GiB), plus Q4\_K\_M / Q5\_K\_M / Q6\_K / Q8\_0 / F16 GGUF builds | |
| 74 | | Post-training | LoRA SFT → checkpoint fusion → WiSE-FT (λ = 0.25) → MGPO reinforcement learning (step 2071) | |
| 75 | | Reasoning format | Emits a `<think>…</think>` block before the answer | |
| 76 | | Evaluated decoding | Greedy, 2048 new tokens, `max_seq_len` 8192 | |
| 77 | | Specialisation | Formal logic: FOL translation, entailment, semantic parsing, Lean formalisation and critique | |
| 78 | | License | webAI Non-Commercial License ver. 1.0 | |
| 79 | |
| 80 | The base model's 65,536-token context is carried through unchanged, but every score on this card |
| 81 | was measured inside an 8,192-token window; longer contexts are inherited rather than validated |
| 82 | here. |
| 83 | |
| 84 | ## Results |
| 85 | |
| 86 | ### Track A — in-domain formal logic |
| 87 | |
| 88 | All arms below were run through the same harness, prompts and decoding settings described under |
| 89 | [Evaluation protocol](#evaluation-protocol). Throughput rows are reported because in-domain score |
| 90 | alone is misleading for a 3B model: `ans/s` is defined throughout as `tok/s ÷ mean generation |
| 91 | length`, so it measures completed answers rather than raw decode rate. |
| 92 | |
| 93 | | lane / metric | TwIL-LM3 | TwIL-LM3* | SmolLM3-3B base | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ | |
| 94 | |---|---:|---:|---:|---:|---:|---:|---:|---:| |
| 95 | | lean_formalize token_f1 | 0.5869 | **0.6456** | 0.4347 | 0.3690 | 0.1321 | 0.4655 | 0.4022 | 0.6306 | |
| 96 | | rule_induction derivation | 0.3192 | **0.9644** | 0.1029 | 0.0825 | 0.0615 | 0.1936 | 0.3680 | 0.6518 | |
| 97 | | entailment_label accuracy | 0.5750 | 0.6867 | 0.3750 | 0.3300 | 0.4700 | 0.5400 | 0.5800 | **0.7750** | |
| 98 | | mcq_answer accuracy | 0.1100 | **0.5200** | 0.0000 | 0.0000 | 0.0150 | 0.0750 | 0.0000 | 0.0700 | |
| 99 | | semantic_parse token_f1 | 0.4416 | **0.8762** | 0.4149 | 0.3102 | 0.3665 | 0.3778 | 0.4257 | 0.4331 | |
| 100 | | lean_critic accuracy | 0.6600 | 0.5200 | 0.6500 | 0.5300 | 0.5900 | 0.5500 | **0.7950** | 0.5550 | |
| 101 | | lm_corpus perplexity ↓ | 2.8972 | 3.1284 | 3.1818 | 2.8478 | 4.3815 | 4.9472 | **2.5440** | 912.23 § | |
| 102 | | math_corpus perplexity ↓ | 3.8229 | **3.5245** | 4.0685 | 4.7531 | 6.7472 | 8.3323 | 4.0083 | 1045.63 § | |
| 103 | | average, 6 lanes | 0.4488 | **0.7021** | 0.3296 | 0.2703 | 0.2725 | 0.3670 | 0.4285 | 0.5192 | |
| 104 | | **macro gate** | 0.4218 | **0.5896** | 0.3466 † | 0.2925 | 0.3473 | 0.3757 | 0.5336 | — | |
| 105 | | **strict-7** | 0.1971 | **0.3290** | 0.1493 | 0.1229 | 0.1579 | 0.1714 | 0.2093 | — | |
| 106 | | macro_primary | 0.4475 | 0.4958 | 0.4075 | 0.3450 | 0.4188 | 0.4213 | **0.5750** | — | |
| 107 | | tok/s | 15880 | 15840 | 15564 | 16160 | **25230** | 22480 | 9420 | 3374 | |
| 108 | | mean gen length | **564** | 572 | 999 | 696 | 2296 | 1830 | 2094 | 1005 | |
| 109 | | **ans/s** | **28.1** | 27.7 | 15.6 | 23.2 | 10.9 | 12.0 | 4.5 | 3.4 | |
| 110 | |
| 111 | \* **TwIL-LM3\*** is our latest version of TwIL-LM3. **The weights will be released soon** — the |
| 112 | files in this repository are the current TwIL-LM3 release, not this one. Lanes marked — are not |
| 113 | yet reported for it. |
| 114 | |
| 115 | ‡ **gpt-oss-120b** runs MXFP4 weights at tensor-parallel 2 — quantized and multi-GPU, so its |
| 116 | throughput rows are not directly comparable to the single-GPU BF16 arms. Its `procedural` lane |
| 117 | and the loose-match scorings were not collected, so the three summary rows below the six-lane |
| 118 | average cannot be computed for it; that is what the — cells mean, not a zero. |
| 119 | |
| 120 | § The 120B's perplexities are three orders of magnitude off every other arm because its harmony |
| 121 | response format and tokenizer make the corpus lanes score a different quantity. The number is |
| 122 | reported for completeness but is not a comparable measurement. |
| 123 | |
| 124 | † The base column here comes from the external-comparison run rather than the paired base-vs-TwIL |
| 125 | run, hence 0.3466 against the 0.3356 quoted in the summary at the top of this card — run-to-run |
| 126 | variation of the same checkpoint. The paired run is the correct basis for the improvement claim. |
| 127 | |
| 128 | **`average, 6 lanes`** is the plain mean of the six objective rows above it, each at whatever |
| 129 | scoring that row reports. It is a coarser summary than the three that follow — it mixes token-F1 |
| 130 | with accuracy — but it is the only summary row every arm here can be compared on, including the |
| 131 | 120B. |
| 132 | |
| 133 | The next three rows aggregate more carefully. None of them include the perplexity lanes or the |
| 134 | token-F1 scorings, which are not on a common 0–1 accuracy scale. |
| 135 | |
| 136 | **`macro gate`** is the headline metric and the one the training pipeline gates on. It is the |
| 137 | equal-weight mean of five objectives: the four bounded classification lanes (`entailment_label`, |
| 138 | `mcq_answer`, `procedural`, `lean_critic`) plus `rule_induction`, scored by its continuous |
| 139 | derivation score. Rule induction is included specifically so a fine-tune cannot pass the gate |
| 140 | while quietly regressing inductive reasoning. In the gate, `mcq_answer` and `procedural` are |
| 141 | credited as `max(exact_match, loose_match)`: for free-text answer lanes, a response that is |
| 142 | correct but differently formatted is a formatting artefact rather than a reasoning failure. This |
| 143 | affects the aggregate only — the per-lane rows above stay strict. |
| 144 | |
| 145 | **`macro_primary`** is the same mean over the four classification lanes alone, without |
| 146 | `rule_induction`. It is the narrower "bounded classification" view, kept for comparability with |
| 147 | earlier reports; the gate is the metric to read for overall in-domain capability. |
| 148 | |
| 149 | **`strict-7`** is the mean of seven lanes scored under strict metrics only (`fol_translation`, |
| 150 | `entailment_label`, `mcq_answer`, `semantic_parse` and `lean_formalize` exact match, |
| 151 | `lean_critic` and `procedural` accuracy), with no loose-match credit anywhere. It is deliberately |
| 152 | harsh — exact match on generative lanes is near zero for every arm — so it is useful for ranking |
| 153 | models against each other but not as an absolute capability measure. |
| 154 | |
| 155 | TwIL-LM3 beats every arm up to and including LFM2.5-8B-A1B, and does so on all six objective |
| 156 | lanes and all four summary rows, not on average alone. Against the strongest of them it is |
| 157 | 0.4218 to 0.3757 on the gate at roughly a third of the total parameters, with the margin coming |
| 158 | from the lanes the pipeline targets directly: `lean_formalize` token-F1 0.5869 against 0.4655, |
| 159 | `rule_induction` 0.3192 against 0.1936, `semantic_parse` 0.4416 against 0.3778. |
| 160 | |
| 161 | It does not beat the two largest arms. Qwen3-8B leads it on the gate 0.5336 to 0.4218 and |
| 162 | gpt-oss-120b leads the six-lane average 0.5192 to 0.4488. That gap is worth reading carefully in |
| 163 | Qwen's case: almost all of it is loose-match credit. Qwen answers MCQ correctly but never in the |
| 164 | requested format — strict accuracy 0.0000 against TwIL-LM3's 0.1100, while its loose match is |
| 165 | 0.745 — and the macro rows credit `max(exact_match, loose_match)`. On `strict-7`, which gives no |
| 166 | loose-match credit anywhere, the two are 0.2093 to 0.1971, a gap of 0.012 rather than 0.11. Qwen |
| 167 | also wins `lean_critic` outright at 0.7950 and has the lowest `lm_corpus` perplexity at 2.5440. |
| 168 | The 120B leads three lanes outright and is genuinely stronger at entailment (0.7750) and rule |
| 169 | induction (0.6518). |
| 170 | |
| 171 | The size and speed context matters for both. Qwen3-8B is 2.6x the parameters and produces 4.5 |
| 172 | answers/sec against TwIL-LM3's 28.1; the 120B is 40x the parameters and produces 3.4. TwIL-LM3 |
| 173 | is the strongest arm here at its own scale and the most efficient arm at any scale. |
| 174 | |
| 175 | The unreleased TwIL-LM3\* moves the gate to 0.5896 and strict-7 to 0.3290, roughly +0.17 and |
| 176 | +0.13 over the current release. The gains are concentrated in the two lanes where TwIL-LM3 is |
| 177 | weakest in absolute terms rather than relative ones — `rule_induction` 0.3192 → 0.9644 and |
| 178 | `semantic_parse` token-F1 0.4416 → 0.8762 — plus strict MCQ accuracy 0.1100 → 0.5200. It gives |
| 179 | back `lean_critic` (0.6600 → 0.5200) and a little `lm_corpus` perplexity, so it is not uniformly |
| 180 | better. |
| 181 | |
| 182 | It is also the most efficient arm in the table by a wide margin — 28.1 answers/sec, from |
| 183 | generations averaging 564 tokens where every other arm except Llama runs past 690. The Liquid |
| 184 | models decode faster in raw tokens per second, 25230 and 22480 against 15880, but their length |
| 185 | more than cancels it. |
| 186 | |
| 187 | ### Track B — held-out benchmarks |
| 188 | |
| 189 | | dataset | TwIL-LM3 | SmolLM3-3B base | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ | |
| 190 | |---|---:|---:|---:|---:|---:|---:|---:| |
| 191 | | gsm8k | 0.8733 | 0.8833 | 0.8300 | 0.8767 | 0.9133 | 0.9567 | **0.9767** | |
| 192 | | svamp | 0.8500 | 0.8567 | 0.8200 | 0.9000 | 0.9133 | **0.9400** | **0.9400** | |
| 193 | | gsm_symbolic | 0.7567 | 0.7633 | 0.8067 | **0.9767** | 0.9267 | 0.8133 | 0.8467 | |
| 194 | | arc_cot | 0.8467 | 0.8400 | 0.7967 | 0.8667 | 0.9033 | 0.9633 | **0.9667** | |
| 195 | | logicbench | 0.7167 | 0.6467 | 0.5733 | 0.6267 | 0.7200 | **0.8567** | 0.8533 | |
| 196 | | strategyqa | 0.6500 | 0.6333 | 0.6533 | 0.6433 | 0.6667 | 0.7400 | **0.7867** | |
| 197 | | drop | 0.7467 | 0.7000 | 0.6733 | 0.6900 | 0.6633 | **0.8833** | 0.8500 | |
| 198 | | csqa | 0.7367 | 0.7067 | 0.7500 | 0.7433 | 0.7700 | **0.8633** | 0.8367 | |
| 199 | | musr | 0.4957 | 0.4997 | 0.4932 | 0.4867 | 0.5703 | 0.6301 | **0.6852** | |
| 200 | | mmlu_redux | 0.6667 | 0.6633 | 0.6000 | 0.7133 | 0.8367 | 0.8500 | **0.9467** | |
| 201 | | ifeval | 0.6433 | 0.6767 | 0.7167 | 0.7300 | **0.8900** | 0.8400 | 0.7900 | |
| 202 | | rudas_ood | 0.0365 | 0.0209 | **0.0733** | 0.0017 | 0.0061 | 0.0468 | 0.0000 ¶ | |
| 203 | | bbh_logic | 0.6633 | 0.6667 | 0.5333 | 0.5713 | 0.7700 | 0.6367 | **0.9980** | |
| 204 | | math500 | 0.6900 | 0.7000 | 0.4233 | 0.7133 | 0.7800 | 0.6100 | **0.8433** | |
| 205 | | **macro (10 CoT datasets)** | 0.7339 | 0.7193 | 0.6997 | 0.7523 | 0.7884 | 0.8493 | **0.8689** | |
| 206 | | **macro (all 14)** | 0.6694 | 0.6612 | 0.6245 | 0.6814 | 0.7378 | 0.7591 | **0.8086** | |
| 207 | | tok/s | 15880 | 15564 | 16160 | 25230 | 22480 | 9420 | 3374 | |
| 208 | | mean gen length | **482** | 626 | 510 | ≈796 | ≈1327 | ≈1931 | 801 | |
| 209 | | **ans/s** | **32.9** | 24.9 | 31.7 | ≈31.7 | ≈16.9 | 4.9 | 4.2 | |
| 210 | |
| 211 | ‡ MXFP4 weights, tensor-parallel 2 — quantized and multi-GPU, so not directly comparable to the |
| 212 | single-GPU BF16 rows. ¶ 74% of its `rudas_ood` generations hit the length cap, so that cell is a |
| 213 | truncation artefact rather than a measured score; excluding the row, its 13-dataset macro is |
| 214 | 0.8708. |
| 215 | |
| 216 | Lengths marked ≈ are derived from stored generations using each model's characters-per-token |
| 217 | ratio rather than re-tokenized directly; the method reproduces the three directly measured |
| 218 | lengths to within 3.5%. |
| 219 | |
| 220 | The honest summary of this table is that TwIL-LM3 does not lead it. Larger models score higher, |
| 221 | in order of size, and the 120B leads nine of fourteen rows. Two things are worth extracting |
| 222 | anyway. First, TwIL-LM3 improves on its own base while sitting mid-table (0.7339 against 0.7193 |
| 223 | on the 10-dataset macro), which is the point of the WiSE-FT stage — in-domain gains without |
| 224 | transfer collapse. Second, it produces the shortest generations of any arm here at 482 tokens |
| 225 | and consequently the most answers per second at 32.9, roughly eight times the 120B's rate. |
| 226 | |
| 227 | ## Usage |
| 228 | |
| 229 | ```python |
| 230 | import torch |
| 231 | from transformers import AutoModelForCausalLM, AutoTokenizer |
| 232 | |
| 233 | model_id = "webAI-Official/TwIL-LM3" |
| 234 | tok = AutoTokenizer.from_pretrained(model_id) |
| 235 | model = AutoModelForCausalLM.from_pretrained( |
| 236 | model_id, torch_dtype=torch.bfloat16, device_map="auto" |
| 237 | ) |
| 238 | |
| 239 | messages = [{"role": "user", "content": |
| 240 | "Does 'All dogs are mammals. Rex is a dog.' entail 'Rex is a mammal'? " |
| 241 | "Answer entailment, contradiction, or neutral."}] |
| 242 | inputs = tok.apply_chat_template( |
| 243 | messages, add_generation_prompt=True, |
| 244 | return_tensors="pt", return_dict=True, |
| 245 | ).to(model.device) |
| 246 | |
| 247 | out = model.generate(**inputs, max_new_tokens=2048, do_sample=False) |
| 248 | print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| 249 | ``` |
| 250 | |
| 251 | `return_dict=True` matters on transformers 5.x, where `apply_chat_template` returns a |
| 252 | `BatchEncoding` rather than a bare tensor; the above works on both 4.x and 5.x. |
| 253 | |
| 254 | The reported numbers use **greedy decoding** (`do_sample=False`) and a **2048-token** generation |
| 255 | budget. Note that the shipped `generation_config.json` inherits SmolLM3's sampling defaults |
| 256 | (`do_sample=true`, `temperature=0.6`, `top_p=0.95`), so `do_sample=False` must be passed |
| 257 | explicitly to reproduce the evaluation. The model opens a `<think>...</think>` reasoning block |
| 258 | before answering, so a short generation budget truncates reasoning and scores far worse. |
| 259 | |
| 260 | ### GGUF / llama.cpp |
| 261 | |
| 262 | Quantized GGUF builds ship in this repository alongside the safetensors weights. The `smollm3` |
| 263 | architecture is supported by llama.cpp, and the chat template, `<|im_end|>` EOS and BOS are |
| 264 | carried into the GGUF metadata, so chat mode works without extra flags. |
| 265 | |
| 266 | | file | quant | size | bits/weight | notes | |
| 267 | |---|---|---:|---:|---| |
| 268 | | `TwIL-LM3-Q4_K_M.gguf` | Q4_K_M | 1.78 GiB | 4.96 | recommended default; runs on CPU or 4 GB of VRAM | |
| 269 | | `TwIL-LM3-Q5_K_M.gguf` | Q5_K_M | 2.06 GiB | 5.74 | a little more headroom than Q4_K_M | |
| 270 | | `TwIL-LM3-Q6_K.gguf` | Q6_K | 2.35 GiB | 6.56 | close to Q8_0 quality at two-thirds the size | |
| 271 | | `TwIL-LM3-Q8_0.gguf` | Q8_0 | 3.05 GiB | 8.50 | near-lossless, for quality-sensitive use | |
| 272 | | `TwIL-LM3-F16.gguf` | F16 | 5.73 GiB | 16.00 | unquantized, for requantization or reference runs | |
| 273 | |
| 274 | ```bash |
| 275 | llama-cli -m TwIL-LM3-Q4_K_M.gguf -cnv --temp 0 -n 2048 |
| 276 | ``` |
| 277 | |
| 278 | Two things matter for reproducing the scores above under llama.cpp. Pass `--temp 0`, because the |
| 279 | evaluation is greedy while the packaged sampling defaults are not. And leave the generation |
| 280 | budget large — 2048 tokens or more — since the model emits a `<think>` block before answering |
| 281 | and a short budget truncates it, which costs far more accuracy than the quantization does. |
| 282 | |
| 283 | F16 and Q8_0 were produced directly by `convert_hf_to_gguf.py` from the released bf16 weights; the |
| 284 | K-quants (Q4_K_M, Q5_K_M, Q6_K) were quantized from the F16 build with `llama-quantize`, without |
| 285 | an importance matrix. All five were smoke-tested for load and generation on CPU. Note that F16 is |
| 286 | not bit-identical to the released weights: bf16 and f16 carry the same 16 bits but trade exponent |
| 287 | range against mantissa precision, so the conversion is a narrowing one, in practice negligible |
| 288 | for inference. |
| 289 | |
| 290 | The published Track A and Track B numbers were measured on the **bf16** weights through vLLM, not |
| 291 | on any of these GGUF builds, so expect small deviations — most likely at Q4_K_M — that have not |
| 292 | been quantified here. |
| 293 | |
| 294 | ## How it was built |
| 295 | |
| 296 | Four stages on top of the base model: |
| 297 | |
| 298 | 1. **LoRA supervised fine-tuning** on a synthetic formal-logic corpus covering the Track A |
| 299 | objectives (first-order-logic translation, entailment labelling, semantic parsing, Lean |
| 300 | formalisation and critique, procedural reasoning, rule induction). |
| 301 | 2. **Checkpoint fusion** — parameter-space averaging of intermediate SFT checkpoints selected |
| 302 | by a diversity probe, rather than taking the final checkpoint. |
| 303 | 3. **WiSE-FT interpolation** toward the pretrained base, `W = (1 − λ)·W_base + λ·W_finetuned` |
| 304 | with **λ = 0.25** — i.e. only a quarter of the fine-tuned delta is retained. λ was chosen by |
| 305 | constrained optimisation: maximise in-domain score subject to minimal degradation on held-out |
| 306 | benchmarks. This conservative λ is the direct reason held-out capability survives. |
| 307 | 4. **MGPO** — entropy-weighted GRPO reinforcement learning against a programmatic verifier, with |
| 308 | partial credit for loose matches and token-F1 so that all-fail prompt groups still produce |
| 309 | gradient. Published checkpoint is **step 2071**. |
| 310 | |
| 311 | A sibling arm that skipped stage 3's conservative interpolation scores considerably higher |
| 312 | in-domain (macro gate 0.515) but gives back roughly twelve points of held-out capability. This |
| 313 | release is the balanced point of that trade; the other was not published. |
| 314 | |
| 315 | ## Limitations and caveats |
| 316 | |
| 317 | **Truncation.** At a 2048-token budget, 4.4% of Track A generations hit the cap — better than |
| 318 | the base's 17.4%, but still above the 2% threshold our protocol requires to mark a comparison |
| 319 | `rankable`. The Track A macro gate should therefore be read as indicative rather than exact. |
| 320 | Because a truncated response scores zero regardless of reasoning quality, both numbers are |
| 321 | pessimistic, and the base substantially more so — meaning the true Track A gap is probably |
| 322 | narrower than +0.086. |
| 323 | |
| 324 | **Scope.** Tuned for formal logic. The Track B suite does not cover code generation or tool use |
| 325 | (HumanEval, LiveCodeBench and BFCL were not run for this model or its base), so this release |
| 326 | makes no claim about those. |
| 327 | |
| 328 | **Not a chat model.** It was optimised against automatic verifiers on logic tasks. It has had no |
| 329 | safety tuning beyond whatever the base model carries, and no instruction-following alignment |
| 330 | work — IFEval regressed slightly. |
| 331 | |
| 332 | **Failed consolidation stage.** A post-RL self-distillation round (SDFT) was attempted and made |
| 333 | both tracks worse at every budget tried (−18% Track A at one epoch on this family). It is not |
| 334 | part of this model. See the accompanying `SDFT_RESULT.md` in the project repository. |
| 335 | |
| 336 | ## Evaluation protocol |
| 337 | |
| 338 | - Track A: `n = 200` per objective, greedy (`temperature = 0`), `max_new_tokens = 2048`, one |
| 339 | retry at 4096 for truncated rows, `max_seq_len = 8192`, seed 42. |
| 340 | - Track B: 300 examples per task, greedy, `max_gen_toks = 4096`, `max_model_len = 8192`, |
| 341 | `repetition_penalty = 1.0`, chat template applied, vLLM backend. |
| 342 | - Both tracks use the same protocol for the model and its base, in a paired run over identical |
| 343 | sampled rows. |
| 344 | |
| 345 | `repetition_penalty = 1.0` is load-bearing. A 1.1 penalty produced apparent 20-point swings on |
| 346 | Track B that were pure decoding artefact; the decoding kwargs are hashed into the protocol |
| 347 | identity so a mismatched runner fails loudly instead of quietly producing a different number. |
| 348 | |
| 349 | ## Relationship to TwIL-LM |
| 350 | |
| 351 | [**TwIL-LM2**](https://huggingface.co/webAI-Official/TwIL-LM) is the 1.7B member of this family, built |
| 352 | from SmolLM2 by the same pipeline. It reaches a higher in-domain score relative to its own base — |
| 353 | and leads every arm we have measured on Track A strict-7, at any size — but it gives back |
| 354 | held-out capability; this model is the one that improves both. Both repositories now ship full |
| 355 | merged models on `main`, loaded directly with `AutoModelForCausalLM`; the original LoRA-adapter |
| 356 | release is archived on that repository's `TwIL-LM1` branch. |
| 357 | |
| 358 | ## License and attribution |
| 359 | |
| 360 | Released under the **webAI Non-Commercial License ver. 1.0** — see `LICENSE.md` in this |
| 361 | repository. |
| 362 | |
| 363 | The base model, [`HuggingFaceTB/SmolLM3-3B`](https://huggingface.co/HuggingFaceTB/SmolLM3-3B), |
| 364 | is Apache 2.0; its licence text is retained as `apache-2.0-LICENSE.txt` and all credit for the |
| 365 | base model goes to the HuggingFaceTB team. Apache 2.0 permits distributing derivative works |
| 366 | under different terms provided attribution is preserved, which is what the pair of licence files |
| 367 | in this repository does. |
| 368 | |