Alireza1913/otel-llm-e4b-it-4bit

OTel-LLM-E4B-IT — 4-bit Quantized

A 4-bit (NF4, double-quantized) version of farbodtavakkoli/OTel-LLM-E4B-IT, re-packaged for low-VRAM inference on consumer and free-tier GPUs (e.g. NVIDIA T4, 15GB).

This repository does not modify the model's weights, training, or behavior in any substantive way — it is a quantized re-packaging intended to make the original OTel-LLM checkpoint runnable on hardware with limited GPU memory. All credit for the underlying model, training data, and evaluation belongs to the original author.

Model Details

Attribute	Value
Base model	google/gemma-4-E4B-it
Source model	farbodtavakkoli/OTel-LLM-E4B-IT
Parameters	4.5B (base)
Quantization	4-bit NF4, double quantization (bitsandbytes)
Compute dtype	float16
Approx. VRAM footprint	~3–4 GB (vs. ~9 GB at bf16/fp16)
Domain	Telecommunications (3GPP, O-RAN, GSMA, IETF RFCs)
Intended use	Context-grounded RAG question answering
Language	English
License	Apache 2.0 (inherits from base)

Why This Repo Exists

The original OTel-LLM-E4B-IT checkpoint requires roughly 9 GB of VRAM at full precision, which exceeds the memory available on free-tier GPUs such as the T4. This repository stores the model quantized to 4-bit precision, allowing it to be loaded and run entirely on GPUs with as little as ~6–8 GB of free VRAM, with no meaningful loss in usable output quality for RAG-style telecom Q&A.

Intended Use

This model is designed for context-grounded telecom answer generation inside Retrieval-Augmented Generation (RAG) pipelines. It expects retrieved context passages alongside the user's question and should not be used as an unrestricted, context-free general knowledge assistant.

In scope:

• Answering telecom questions (3GPP, O-RAN, GSMA specifications, IETF RFCs) when relevant context is supplied
• RAG pipelines built around telecom knowledge bases
• Research and prototyping on telecom-domain LLM applications

Out of scope:

• General-purpose chat or reasoning outside the telecom domain
• Unrestricted, context-free QA (use the base OTel model or a dedicated -Safety variant instead)
• Safety-critical, regulatory, or customer-facing decisions without human verification

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "YOUR-USERNAME/otel-llm-e4b-it-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    low_cpu_mem_usage=True,
)  # quantization_config is stored in this repo's config and applied automatically

prompt = """You are a precise telecom assistant in a RAG pipeline.
Use only the retrieved context to answer.

User Question
What is the purpose of the F1 interface in O-RAN?

Retrieved Contexts
CONTEXT 1
The F1 interface connects the O-RAN Distributed Unit (O-DU) to the O-RAN Central Unit (O-CU).

Answer:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Requirements: transformers, accelerate, bitsandbytes, sentencepiece

Limitations

• Inherits all limitations of the base OTel-LLM-E4B-IT model: English-only, telecom-domain-specific, and evaluated on held-out data from the same distribution as training (not an independent external benchmark).
• 4-bit quantization introduces a small, generally negligible accuracy trade-off versus the full-precision checkpoint; it has not been independently re-benchmarked in this repository.
• Generated content should be verified by a domain expert before any operational, regulatory, safety-related, or customer-facing use.
• Not intended for context-free, general-purpose question answering.

Attribution & License

• Original model: farbodtavakkoli/OTel-LLM-E4B-IT by Farbod Tavakkoli, part of the OTel Family of Models
• Base model: google/gemma-4-E4B-it
• License: Apache 2.0, inherited from the upstream release. Users must comply with both the OTel release license and the Gemma base model's terms of use.

This is an unofficial, community-produced quantization and is not affiliated with or endorsed by the original OTel-LLM authors.

Citation

If you use this model, please cite the original work:

@misc{otel_models_2026,
  title  = {OTel: Open Telco AI Datasets, Benchmarks, and Models},
  author = {Tavakkoli, Farbod and others},
  year   = {2026},
  note   = {Open Telco (OTel) model release},
  url    = {https://huggingface.co/farbodtavakkoli}
}