SixVolts/GLM-5.2-ewaste-edition-GGUF

GLM-5.2 — "e-waste edition" GGUF

Imatrix GGUF quantizations of GLM-5.2 (745B total / ~40B active, glm-dsa arch, DeepSeek-style MoE with 256 routed + 1 shared expert, 8 active/token, MLA attention) tuned for MoE inference on older / "e-waste" hardware — pre-AVX-512 CPUs and previous-gen datacenter GPUs (MI100 / gfx908).

The guiding choice across every build (Q4_K / Q3_K / Q2_K) is K-quants for the routed experts, not codebook i-quants (IQ2/IQ3): K-quant dequantization is dramatically faster on pre-AVX-512 CPUs and on gfx908, where expert dequant — not bandwidth — is the decode bottleneck. An i-quant that looks smaller on paper is slower to run on this class of hardware. (The experimental Q2_K_XL build keeps the same principle — experts stay Q2_K — except for its 13 coldest, rarely-fired layers; see below.)

Available quantizations

Variant	Size	Routed experts	Non-experts	Target
GLM-5.2-Q4_K_XL	403.8 GiB	Q4_K	Q8_0	Highest quality; big-RAM CPU-expert boxes (≈512 GB). Too big for 10×32 GB 0-spill
GLM-5.2-Q3_K_XL	312.9 GiB	Q3_K	Q8_0	Max quality on GPU; experts spill to CPU on <11×32GB
GLM-5.2-Q3_K_M ⭐ recommended	295.7 GiB	Q3_K + Q2_K (cold layers)	Q6_K	Fits 0-spill on 10×32 GB (320 GB HBM)
GLM-5.2-Q2_K_XL	226.9 GiB	Q2_K + IQ2_XXS (13 coldest layers)	Q4_K	Fits 0-spill on 8×32 GB (256 GB HBM); fastest decode, lowest quality

All are imatrix-quantized. Q3_K_M is the recommended build: 17 GiB smaller than Q3_K_XL at near-identical quality (+0.6% perplexity), exactly enough to keep the entire model resident across ten 32 GB MI100s with zero CPU spill — eliminating the expert-dequant decode bottleneck the whole project exists to fight.

Q2_K_XL trades quality for a much smaller footprint: at 226.9 GiB it fits 0-spill across eight 32 GB cards (256 GB HBM) where Q3_K_M needs ten, and because it moves fewer bytes per token it actually decodes faster than Q3_K_M (14.7 vs 13.2 tok/s, both 0-spill). It drops non-expert infra to Q4_K, keeps experts as fast Q2_K k-quants (only the 13 imatrix-coldest layers go to IQ2_XXS), 2.59 bpw. The trade is real quality loss — see the perplexity table — so it's for rigs that are HBM-limited to eight cards, or that want maximum decode throughput and can tolerate Q2-level quality. Run it with -sm layer (not -sm tensor).

---

GLM-5.2-Q4_K_XL — composition

The simplest recipe in the set and the highest quality: every routed expert at Q4_K, everything else at Q8_0. It's the direct one-tier-up sibling of Q3_K_XL (identical recipe, q3_K→q4_K on the experts, same wikitext imatrix), built for big-RAM CPU-expert inference — a box with ~512 GB where the routed experts live in system RAM and a single modest GPU (or none) handles the rest.

Tensor group	Quant
Routed experts (ffn_{gate,up,down}_exps), all MoE layers incl. blk.78 (MTP head)	Q4_K
MLA attention (q_a, q_b, kv_a_mqa, k_b, v_b, output) + indexer	Q8_0
Shared expert (ffn_{gate,up,down}_shexp) — fires every token	Q8_0
token_embd, output, dense FFN (blk.0–2)	Q8_0
norms, router (ffn_gate_inp, exp_probs_b)	F32

Total: 403.8 GiB, 4.60 bpw, 10 shards. imatrix-quantized (wikitext-2, 200 × 512-tok) applied to the experts — the same importance matrix as Q3_K_XL. Keeping every non-expert tensor at Q8_0 (rather than the Q6_K the GPU-fit builds use) spends bytes on quality, since a 512 GB host has the room. Fits a 512 GB box with headroom for a long KV cache; too large for 10×32 GB GPU 0-spill — use Q3_K_M for that.

---

GLM-5.2-Q3_K_M — composition

Built from the unsloth BF16 of GLM-5.2 (unsloth/GLM-5.2-GGUF) using unsloth's importance matrix (see imatrix note below). Base ftype Q3_K_M, with explicit per-tensor overrides:

Tensor group	Quant	Notes
Routed experts, blk.35–77 (43 hottest MoE layers)	Q3_K	Q3_K_M base; some late-layer ffn_down_exps auto-promoted to Q4_K
Routed experts, blk.3–34 (32 coldest MoE layers)	Q2_K	the cold cut — see below
Routed experts, blk.78 (MTP / nextn head)	Q2_K	head is unused in normal decode
Attention (q_a, q_b, kv_a_mqa, v_b, output)	Q6_K	down from Q8_0
Shared expert (ffn_{gate,up,down}_shexp)	Q6_K	fires every token
token_embd, output	Q6_K
attn_k_b	Q4_0	ncols=192 not divisible by 256 → automatic fallback (harmless)

Total: 295.71 GiB, 3.37 bpw, 8 shards.

Which layers were dropped to Q2_K, and why

Only the routed experts of the 32 coldest MoE layers (blk.3–blk.34) were dropped to Q2_K (plus the unused blk.78 MTP head). Blocks 0–2 are dense (no routed experts); blocks 35–77 — the hottest 43 MoE layers — stay at full Q3_K.

"Cold" is measured directly from unsloth's imatrix in_sum2 (the sum of squared activations per input channel — an importance signal). The mean ffn_down_exps importance is monotonic in depth and spans ~6 orders of magnitude:

blk.3   ≈ 0.003      ← coldest
blk.20  ≈ 2.3
blk.34  ≈ 75
blk.39  ≈ 140
blk.77  ≈ 16022      ← hottest

The coldest 32 layers all sit below importance ≈ 75; the hot tail kept at Q3_K runs 88 → 16,022. Because the early layers carry 100–1000× less activation energy than the late ones, dropping their experts to 2-bit costs almost nothing in quality (measured below) while reclaiming the ~17 GiB needed to fit ten cards. The MoE granularity constraint (one quant type per expert tensor — all 256 experts of a layer share it) means the cut is per-(layer, projection); the imatrix already protects the most-used experts within each tensor.

---

GLM-5.2-Q2_K_XL — composition

Same BF16 source and unsloth imatrix. Built to cross under the 256 GB (8 × 32 GB) HBM budget at 0-spill, so every byte was cut to the floor that still runs as fast K-quants on gfx908:

Tensor group	Quant	Notes
Routed experts, all MoE layers except the cold cut	Q2_K	2-bit k-quant — still fast to dequant, unlike i-quants
Routed experts, 13 imatrix-coldest layers (blk.3,4,5,6,7,9–16)	IQ2_XXS	the only i-quant tier; these layers carry ~100–1000× less activation energy
Routed experts, blk.78 (MTP / nextn head)	IQ2_XXS	unused in normal decode
Attention (q_a, q_b, kv_a_mqa, v_b, output) + indexer	Q4_K	down from Q6_K
Shared expert (ffn_{gate,up,down}_shexp)	Q4_K	fires every token
token_embd, output	Q4_K
attn_k_b	Q4_0	ncols=192 not divisible by 256 → automatic fallback (harmless)

Total: 226.9 GiB, 2.59 bpw, 6 shards. The IQ2_XXS layers are the one place this build breaks the "K-quants only" rule — but only on the 13 coldest layers, where the dequant cost is paid rarely and the 2.06-bpw saving is what lets the model clear the 8-card budget.

---

Quality (perplexity)

wikitext-2, 100 chunks @ 512 ctx, -fa on -ctk q8_0 -ctv q8_0, identical settings for all three:

Quant	Size	bpw	PPL ↓	vs this quant
GLM-5.2-Q4_K_XL	403.8 GiB	4.60 / Q8	2.6733	best quality (−5.7% vs Q3_K_M, +108 GiB)
GLM-5.2-Q3_K_XL	312.9 GiB	~3.4 / Q8	2.8176	−0.6% (reference, +17 GiB)
GLM-5.2-Q3_K_M	295.7 GiB	3.37	2.8348	—
unsloth UD-IQ3_XXS	262.3 GiB	~3.06	3.2635	+15% (−33 GiB, i-quant)
GLM-5.2-Q2_K_XL	226.9 GiB	2.59	3.6129	+27% (−69 GiB)

• Q4_K_XL is the quality ceiling of the family: at 2.6733 it sits 0.14 below Q3_K_XL (−5% perplexity) — the payoff for taking the routed experts Q3_K → Q4_K. It costs +91 GiB over Q3_K_XL and won't fit ten 32 GB cards 0-spill; it's for big-RAM CPU-expert hosts (≈512 GB) that want the best quality this set offers.
• vs the full-quality Q3_K_XL: +0.6% perplexity for −17 GiB — near-free, and the 17 GiB is what enables 0-spill on 10 cards.
• vs unsloth's similarly-positioned UD-IQ3_XXS: this quant is 13% lower perplexity while also being a K-quant (far faster to dequant on gfx908). It spends 33 GiB more to get there — a deliberate trade for this hardware class.
• Q2_K_XL is the quality floor: +27% perplexity vs Q3_K_M (and ~+11% vs UD-IQ3_XXS, which it undercuts by 35 GiB and in dequant speed). It exists to buy the 8-card 0-spill fit and the fastest decode — pick it only when those matter more than quality. For best quality-per-byte, use Q3_K_M.

(Full 200-chunk perplexity for Q3_K_M = 3.1408 ± 0.029; the 100-chunk figures above are the matched cross-quant comparison. Q4_K_XL = 2.6733 ± 0.033 over 100 chunks.)

---

Performance — 10 × MI100 (gfx908), 0-spill

-sm layer -fit on -fa on -ctk q8_0 -ctv q8_0 -c 8192, all 295.7 GiB resident on GPU (298.9 GiB total VRAM incl. KV + compute; zero CPU expert spill):

	Q3_K_M (this)
Decode	13.2 tok/s
Prefill (3.5k-token prompt)	101 tok/s default → 147 tok/s at -ub 2048
CPU expert spill	0

For contrast, Q3_K_XL at 312.9 GiB does not fit the same ten cards — it spills ~12 GiB of experts back to CPU, and that spilled expert dequant is the decode bottleneck. The whole point of Q3_K_M is to cross under the HBM budget so nothing spills.

-ub 2048 is a free +45% on prefill here purely because the 0-spill fit leaves ~20 GiB of VRAM headroom to spend on a bigger compute batch.

GLM-5.2-Q2_K_XL — 8 × MI100 (gfx908), 0-spill

-sm layer -fit on -fa on -ctk q8_0 -ctv q8_0, all 226.9 GiB resident across eight 32 GB cards (zero CPU spill):

	Q2_K_XL
Decode	14.7 tok/s — faster than Q3_K_M's 13.2 (fewer bytes/token)
Decode + ngram-simple	up to 21.9 tok/s on repetitive / templated output (accept 1.0)
CPU expert spill	0

Q2_K_XL is the throughput pick: it fits fewer cards (8 vs 10) and decodes faster, paying for it in quality (perplexity table above).

Speculative decoding — use ngram-simple, not ngram-cache

Model-free n-gram self-speculation, measured on the 0-spill 10×MI100 setup (decode tok/s):

--spec-type	prose	structured code	verbatim repeat
(none)	13.1	13.0	13.1
ngram-simple ✅	13.1	15.9 (+22%)	16.6 (+27%)
ngram-cache	8.8 (−33%)	10.3 (−21%)	17.9 (+37%)

Use ngram-simple — it is safe to leave on. It only drafts on a high-confidence n-gram match (accept rate ≈ 1.0 when it fires), so it abstains on unpredictable prose with zero overhead and speeds up structured / repetitive output. Draft depth (--spec-draft-n-max) barely matters here — accept is already ~1.0, so 1–4 are within noise.

Tuning repetition coverage — --spec-ngram-simple-size-n (default 12). The default only fires on a confident 12-token match, so it catches tight repetition (it abstains on looser patterns). Lower it to fire more aggressively, at the risk of mis-drafting semi-structured text. Measured on Q2_K_XL (decode tok/s):

size_n	prose	log-line / templated repeat	free-form code
12 (default, safe)	14.7 (abstain)	14.5 (abstain)	14.6 (abstain)
8 (moderate)	14.6 (abstain)	19.7 (+36%)	12.6 (−14%)
6 (aggressive)	14.7 (abstain)	20.2 (+37%)	7.1 (−51%)

Prose stays neutral at every setting. Keep 12 for mixed / code-heavy serving; drop to 8 for agentic / log / templated workloads. size_n=6 over-drafts repeating code structure (def get_X with varying field names → 3% accept) and is only worth it on known highly-repetitive output.

Avoid ngram-cache for general serving: it drafts aggressively on weak matches (accept ~0.13 prose / ~0.37 code), and on this MoE every rejected draft still routes its 8 experts through the verify batch, so the wasted expert-GEMM costs more than it saves except on near-verbatim echo. Why spec is expensive on MoE at all: the verify batch routes every drafted position to its own experts, so wasted drafts cost real GPU compute — speculation here is gated by accept rate, not by spill.

---

Usage

# Q4_K_XL — big-RAM CPU-expert box (≈512 GB). Routed experts stay in system RAM;
# attention + KV + shared expert go on one modest GPU. Loads the 10-shard set from the first shard.
llama-server -m GLM-5.2-Q4_K_XL-00001-of-00010.gguf \
  -ngl 999 -ot 'ffn_.*_exps=CPU' -fa on -ctk q8_0 -ctv q8_0 -c 16384
# no GPU? use -ngl 0 for all-CPU. On dual-socket hosts prefix with `numactl --interleave=all`.

# Q3_K_M — llama.cpp (mainline). Loads the 8-shard set from the first shard.
llama-server -m GLM-5.2-Q3_K_M-00001-of-00008.gguf \
  -sm layer -fit on -fa on -ctk q8_0 -ctv q8_0 -c 8192 -ub 2048 \
  --spec-type ngram-simple        # safe always-on; +20-27% on structured/repetitive, 0 on prose

# Q2_K_XL — fits 0-spill on eight 32 GB cards. Loads the 6-shard set from the first shard.
HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
llama-server -m GLM-5.2-Q2_K_XL-00001-of-00006.gguf \
  -sm layer -fit on -fa on -ctk q8_0 -ctv q8_0 -c 8192 \
  --spec-type ngram-simple        # for log/templated workloads add: --spec-ngram-simple-size-n 8

Q3_K_M needs ~296 GiB of combined HBM for 0-spill (e.g. 10 × 32 GB); Q2_K_XL needs ~228 GiB (e.g. 8 × 32 GB). With less, -fit auto-spills the coldest experts to CPU and it still runs — just slower on the spilled layers. Use -sm layer (not -sm tensor) for both.

Credits / provenance

• Base model: GLM-5.2 by zai-org.
• Source weights (BF16) and the importance matrix used for this quant are from unsloth/GLM-5.2-GGUF — the imatrix is not redistributed here; grab it from unsloth's repo if you want to reproduce. Thanks to the unsloth team for the calibration data.
• Quantized with llama.cpp llama-quantize (--imatrix + --tensor-type-file).
• Provenance differs by build: Q3_K_M / Q2_K_XL use the unsloth BF16 + unsloth imatrix above. The _XL builds (Q3_K_XL, Q4_K_XL) instead use a wikitext-2 imatrix and a plain convert_hf_to_gguf.py → Q8_0 intermediate, with base type Q8_0 and experts overridden via --custom-q 'ffn_.*_exps=q{3,4}_K' — i.e. everything Q8_0 except the routed experts.