This quant needs fixing
config.json's ignore is missing three entries present in the model card's creation snippet, and 86 weight_scale tensors exist for modules the recipe excludes, which trips assert key == "w2_weight" in vLLM's sink_experts loader.
their creation script completely skipped the expert weights entirely.
If you look closely at their exact regex choices inside the ignore block, they bypassed the Mixture of Experts (MoE) layers accidentally:
python
ignore=[
...
"re:.gate.", # <--- This is the culprit
...
]
The Wildcard Over-Match
The Red Hat team intended to skip the linear routing gates (e.g., model.layers.X.mlp.gate) to preserve MoE router accuracy. However, by using the aggressive string wildcard "re:.gate.", they inadvertently instructed llm-compressor to skip any parameter path containing the letters "gate".
In many advanced MoE configurations (including variants of Inkling), the fused w13_weight layer is internally mapped or registered as part of the gate-up projection block (representing the gate_proj and up_proj matrices squeezed together). Because the underlying code framework references the word "gate" when traversing the layer indices for w13, Red Hat's script completely ignored those layers.
The Downstream Consequence
Because their script skipped the layers, llm-compressor never attempted to quantize w13_weight or w2_weight in their run.