README.md
7.8 KB · 187 lines · markdown Raw
1 ---
2 license: apache-2.0
3 language:
4 - en
5 base_model:
6 - google/gemma-4-E4B-it
7 datasets:
8 - farbodtavakkoli/OTel-LLM
9 tags:
10 - telecom
11 - telecommunications
12 - gsma
13 - rag
14 - full-parameter-fine-tuning
15 - fine-tuned
16 pipeline_tag: text-generation
17 ---
18
19 # OTel-LLM-E4B-IT
20
21 **OTel-LLM-E4B-IT** is a context-grounded telecom language model full-parameter fine-tuned on OTel telecommunications data. It is part of the [OTel Family of Models](https://huggingface.co/collections/farbodtavakkoli/otel-llm), an open-source initiative to build reference AI resources for the global telecommunications sector.
22
23 Across the core OTel LLM baselines, OTel fine-tuning improves context-grounded correctness over the base checkpoints by +3.7 to +10.0 percentage points.
24
25 ## Community Use
26
27 As of June 23, 2026, the released OTel models had more than 18 million downloads, and the Open Telco AI project had received 157+ pieces of media coverage worldwide.
28
29 ## Model Details
30
31 | Attribute | Value |
32 |---|---|
33 | Base model | [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) |
34 | Parameters | 4.5B |
35 | OTel training dataset | [OTel-LLM](https://huggingface.co/datasets/farbodtavakkoli/OTel-LLM) |
36 | Dataset fields | `prompt`, `completion`, `abstention`, chunk-count metadata, token-count metadata |
37 | Training method | Full-parameter post-training / fine-tuning |
38 | Language | English |
39 | OTel release license | Apache 2.0 |
40
41 ## Model Lineage
42
43 `google/gemma-4-E4B-it` -> `OTel-LLM` full-parameter post-training -> `farbodtavakkoli/OTel-LLM-E4B-IT`
44
45 ## OTel vs. Base Model
46
47 | Metric | Base model | OTel fine-tuned | Delta | Evaluation split |
48 |---|---:|---:|---:|---|
49 | LLM-as-judge correctness | 82.4% | 91.7% +/- 0.4 | +9.3 pp | OTel-LLM held-out 10% |
50
51 Standard errors are computed with bootstrap resampling (`n=10`) over the held-out OTel evaluation partition. LLM correctness is judged by GPT-4o mini using the retrieved context and reference answer.
52
53 ## Evaluation Caveats
54
55 - LLM results measure context-grounded answer generation from retrieved context, not unrestricted context-free telecom QA.
56 - Reported standard errors come from bootstrap resampling over the held-out evaluation partitions.
57 - Answer quality depends on the retriever, reranker, context window, and prompt policy around the model.
58 - External benchmark transfer, multilingual performance, and per-subdomain performance should be evaluated separately for production settings.
59
60
61 ## Training Data
62
63 The model was trained on telecom-focused data curated by 100+ domain experts. The raw corpus contained roughly 1.1M training points and was filtered to 326,767 higher-confidence examples.
64
65 | Source | Contributor |
66 |---|---|
67 | arXiv telecom papers, 3GPP standards, telecom Wikipedia, telecom Common Crawl | Yale University |
68 | GSMA Permanent Reference Documents, Discover portal | GSMA |
69 | IETF RFC series | NetoAI |
70 | Industry whitepapers | Khalifa University |
71 | O-RAN specifications (working groups 1, 2, 4, 5, 6, 7, 8, 9, 10) | University of Leeds |
72 | O-RAN documents across working groups | The University of Texas at Dallas |
73
74 Released datasets: [OTel-LLM](https://huggingface.co/datasets/farbodtavakkoli/OTel-LLM), [OTel-Embedding](https://huggingface.co/datasets/farbodtavakkoli/OTel-Embedding), [OTel-Reranker](https://huggingface.co/datasets/farbodtavakkoli/OTel-Reranker), and [OTel-Safety](https://huggingface.co/datasets/farbodtavakkoli/OTel-Safety).
75
76 The OTel datasets release derived QA/retrieval/reranking examples rather than the raw source documents.
77
78 Each released dataset includes a dataset card and Croissant metadata with Responsible AI fields for data limitations, biases, sensitive-information considerations, use cases, social impact, synthetic-data status, and provenance.
79
80 ## Representative Training Row
81
82 `OTel-LLM` rows pair a context-grounded telecom RAG prompt with a reference completion.
83
84 ```json
85 {
86 "anchor": "How can a cell be considered to be operating in MBSFN mode for 3.84/7.68 Mcps TDD?",
87 "completion": "A cell shall be considered to be operating in MBSFN mode when individual scrambling codes are assigned to all timeslots via the IE \"TDD MBSFN Information\".",
88 "abstention": false,
89 "n_positive_chunks": 1,
90 "n_negative_chunks": 4
91 }
92 ```
93
94 ## Intended Use
95
96 This model is intended for context-grounded telecom answer generation in Retrieval-Augmented Generation (RAG) pipelines. It should receive retrieved telecom context and generate an answer grounded in that context.
97
98 The model is not optimized for unrestricted context-free question answering. For questions where the retrieved context is missing or insufficient, use an abstention-aware prompt or one of the dedicated `-Safety` variants.
99
100 ## Training Recipe
101
102 | Item | Value |
103 |---|---|
104 | Framework | ScalarLM |
105 | Optimizer | AdamW, 8-bit |
106 | Learning-rate schedule | Cosine decay with warmup |
107 | Weight decay | 0.01 |
108 | Warmup steps | 100 |
109 | Random seed | 42 |
110 | Maximum sequence length | 1500 tokens |
111 | Precision | BF16 |
112 | Attention | Flash Attention 2 |
113 | Distributed training | Fully Sharded Data Parallel |
114 | Gradient checkpointing | Enabled |
115 | Epochs | 3 for LLM/embedding models; 2 for rerankers |
116 | Compute | AMD MI300X/MI325X/MI355X and NVIDIA A100/H100 GPUs |
117
118 ## Usage
119
120 ```python
121 from transformers import AutoModelForCausalLM, AutoTokenizer
122 import torch
123
124 model_name = "farbodtavakkoli/OTel-LLM-E4B-IT"
125 tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
126 model = AutoModelForCausalLM.from_pretrained(
127 model_name,
128 torch_dtype=torch.bfloat16,
129 device_map="auto",
130 trust_remote_code=True,
131 )
132
133 prompt = """You are a precise telecom assistant in a RAG pipeline.
134 Use only the retrieved context to answer.
135
136 User Question
137 What is the purpose of the F1 interface in O-RAN?
138
139 Retrieved Contexts
140 CONTEXT 1
141 The F1 interface connects the O-RAN Distributed Unit (O-DU) to the O-RAN Central Unit (O-CU).
142
143 Answer:"""
144
145 inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
146 outputs = model.generate(**inputs, max_new_tokens=256)
147 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
148 ```
149
150 ## Limitations and Responsible Use
151
152 - OTel models are domain-specific to telecommunications and should not be treated as general-purpose models.
153 - The current release is English-only and primarily text-centric.
154 - The reported OTel performance results use held-out OTel evaluation partitions and should not be interpreted as results from a fully independent external benchmark suite.
155 - Aggregate scores can hide subdomain variation; collaborator stress tests suggest O-RAN retrieval is comparatively strong, while academic-paper and GSMA PRD examples need further curation.
156 - Generated telecom content should be verified before operational, customer-facing, regulatory, safety, or network-configuration use.
157 - Users must comply with both the OTel release license and the upstream base-model license or terms.
158 - For unrestricted telecom QA without retrieved context, use a separately evaluated context-free QnA model rather than assuming this RAG-oriented checkpoint will behave optimally.
159
160 ## Related Models
161
162 - [OTel LLM Collection](https://huggingface.co/collections/farbodtavakkoli/otel-llm)
163 - [OTel Embedding Collection](https://huggingface.co/collections/farbodtavakkoli/otel-embedding)
164 - [OTel Reranker Collection](https://huggingface.co/collections/farbodtavakkoli/otel-reranker)
165
166 ## Project Resources
167
168 - Project page: https://huggingface.co/farbodtavakkoli
169 - Code: https://github.com/farbodtavakkoli/OTel
170 - Media coverage list: https://github.com/farbodtavakkoli/OTel/blob/main/docs/media_coverage.md
171
172 ## Citation
173
174 ```bibtex
175 @misc{otel_models_2026,
176 title = {OTel: Open Telco AI Datasets, Benchmarks, and Models},
177 author = {Tavakkoli, Farbod and others},
178 year = {2026},
179 note = {Open Telco (OTel) model release},
180 url = {https://huggingface.co/farbodtavakkoli}
181 }
182 ```
183
184 ## Contact
185
186 For technical questions, contact farbod.tavakkoli@att.com or farbodtavakoli@gmail.com.
187