Qwythos-9B-Claude-Mythos-5-1M-optiq-5bpw-mlx

MLX quantization of empero-ai/Qwythos-9B-Claude-Mythos-5-1M for Apple Silicon.

Note — text tower only. The source model is a Qwen3.5-VL multimodal model (Qwen3_5ForConditionalGeneration, with a vision encoder). This MLX conversion contains only the text/language tower — the vision encoder weights are not included, so this is a text-only model and does not accept image or video input. The text reasoning the original is benchmarked for (GSM8K, MMLU) is unaffected.

It loads via the standard MLX LLM path (mlx-lm, LM Studio). For LM Studio compatibility the config carries partial_rotary_factor inside rope_parameters (LM Studio's engine hard-indexes that key, unlike mlx-lm which defaults it); the config is also tagged as a causal LM (architectures: ["Qwen3_5ForCausalLM"], vision/image/video token ids removed) to reflect that it is text-only.

Variant: OptiQ mixed-precision (target 5.0 bpw)
Disk size: 6621 MB
Quantized by: sahilchachra

About this quantization

Unlike uniform 4-bit quantization (which forces every layer onto the same bit grid and often collapses reasoning at low bit widths), this model was quantized with mlx-optiq using per-layer KL-sensitivity analysis:

A small calibration set (32 samples spanning prose, multi-step reasoning, code, and constraint-following instructions) is run through the FP16 reference and through trial quantizations of each layer.
The output drift per layer is measured. Layers whose outputs are most affected by quantization (typically the final attention projections, the lm_head, and a few middle blocks) get more bits; layers that tolerate aggressive quantization get fewer.
The final assignment hits the target average bits-per-weight while keeping the bits where they matter. This trades off precision unequally so the average comes out near the target (5.0 bits/weight), but the bits that matter most for output fidelity stay high.

Quantization config

Method: optiq_mixed_precision (mlx-optiq)
Target bits/weight: 5.0
Achieved bits/weight (quantized linear layers): 5.002
Effective bits/weight (whole model): ~6.19 — the 5.0 bpw target applies to the quantized linear layers. The token embedding and lm_head (≈2B params over a 248k-token vocab, with tie_word_embeddings=false) are kept in bf16, which raises the whole-model average and is why the on-disk size (6621 MB) exceeds what a uniform 5-bit model would be.
Candidate bits: [4, 6, 8]
Group size: 64
Sensitivity reference: uniform_4bit
Calibration: 32-sample 4-domain mix (prose + reasoning + code + constraints)

Per-layer bit allocation

248 quantizable components total. OptiQ allocated bits non-uniformly based on KL sensitivity:

Bits	Components	Share
8-bit	56	22.6%
6-bit	111	44.8%
4-bit	81	32.7%
Total	248	100.0%

Benchmark results

Evaluated on Apple M5 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

	This model	FP16 baseline
Decode tok/s (avg, long traces)	43.67	N/A
Peak memory (GB)	7.367	N/A
Disk size (MB)	6621	17969

Quality

Benchmark	This model	FP16 baseline	n
GSM8K (math, accuracy)	94.0%	N/A	50
MMLU (knowledge, accuracy)	80.0%	N/A	50

Context scaling (decode tok/s)

Context length	Decode tok/s
~128 tokens	44.9
~256 tokens	45.0
~512 tokens	44.9
~1024 tokens	44.9

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/Qwythos-9B-Claude-Mythos-5-1M-optiq-5bpw-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=256, verbose=True)

All variants in this collection

Model	Variant
sahilchachra/Qwythos-9B-Claude-Mythos-5-1M-mxfp4-mlx	Block float MX FP4
sahilchachra/Qwythos-9B-Claude-Mythos-5-1M-mxfp8-mlx	Block float MX FP8
sahilchachra/Qwythos-9B-Claude-Mythos-5-1M-optiq-5bpw-mlx	OptiQ mixed-precision (target 5.0 bpw) ← this model

Notes

Requires Apple Silicon (M1 or later) with MLX
Benchmarks run on Apple M5 Pro, 24 GB unified memory
License: see empero-ai/Qwythos-9B-Claude-Mythos-5-1M for the original model's license