Ornith-1.0-35B-EXL3-4.0bpw

EXL3 quantization of deepreinforce-ai/Ornith-1.0-35B, at 4.0 bits per weight (--hq enabled, effective ~4.08 bpw excluding the head).

Why 4.0 bpw?

The source is a ~35B-total Mixture-of-Experts model (qwen3_5_moe, 256 experts / 8 active). At 4.0 bpw the weights occupy roughly 18–20 GB, which fits a single 24 GB GPU (e.g. RTX 3090/4090) with room for context — unlike higher bitrates which exceed 24 GB.

Details

• Format: EXL3 (exllamav3), measurement + quantization in one pass with --hq.
• Tool: exllamav3 v0.0.43, PyTorch 2.8.0 + CUDA 12.8, on an RTX 3090.
• Embeddings / norms: kept at FP16; lm_head at ~6 bpw.
• Vision tower: the model is multimodal (image+video); the vision encoder is kept at original precision and copied through. (exllamav3 v0.0.43 required a one-line patch to whitelist the qwen3_5_moe_vision vision config model_type.)
• MTP head excluded: the base config declares mtp_num_hidden_layers = 1, but the repository ships no mtp.* weights, so the optional multi-token-prediction (speculative-decoding draft) head is omitted. This does not affect the main model.

Usage

Load with exllamav3, or serve via TabbyAPI.

from exllamav3 import Model, Config, Cache, Tokenizer
model = Model.from_config(Config.from_directory("/path/to/Ornith-1.0-35B-EXL3-4.0bpw"))

Calibration

Quantized using exllamav3's built-in calibration dataset (a broad text mix), default rows/length.

---

Quantized with exllamav3. Original model © DeepReinforce AI — see the base model card for its license and terms.