tashfene/Ornith-1.0-35B-MTP-Q4_K_M-GGUF

Ornith-1.0-35B MTP-graft Q4_K_M (GGUF)

The only Ornith MTP build that fits 24 GB cards. Ornith-1.0-35B (Q4_K_M) with the multi-token-prediction head from Qwen3.6-35B-A3B-MTP grafted on, enabling self-speculative decoding in llama.cpp.

• 22.8 GB — fits Tesla P40 / RTX 3090 / RTX 4090 fully on-GPU with 128k context (q8_0 KV cache)
• +39% code generation, +27% prose measured vs plain Ornith Q4_K_M
• Output-exact: speculative decoding never changes results — Ornith itself verifies every drafted token, so quality is identical to the base model by construction. A mismatched head could only cost speed; benchmarks show it doesn't.

Existing MTP grafts (Q6_K 29 GB, Q8_0 38 GB) don't fit 24 GB cards, which is what motivated this build.

Usage

Requires llama.cpp b9859 (2026-07-01) or newer.

llama-server -m ornith-1.0-35b-MTP-graft-Q4_K_M.gguf \
  -ngl 99 -c 131072 -ub 1024 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -fa on --spec-type draft-mtp --spec-draft-n-max 2 \
  --jinja

--spec-draft-n-max 2 is optimal; deeper drafts lose speed (see sweep below).

Benchmarks (Tesla P40, llama.cpp b9859, 200W cap)

Task	Plain Ornith Q4_K_M	MTP graft	Speedup
Code generation (400 tok, temp 0.3)	56.1 tok/s	77.9 tok/s	+39%
Prose generation (300 tok, temp 0.7)	~56 tok/s	71.2 tok/s	+27%

Draft-depth sweep (code / prose tok/s): n-max 1 → 71.2 / 72.1, n-max 2 → 77.9 / 71.2, n-max 3 → 75.7 / 62.4.

VRAM with q8_0 KV: 22.7 GB at 64k context, 23.9 GB at 128k (the practical max on a 24 GB card — the MTP layer needs its own compute buffers). Generation at 17k-deep context: 58 tok/s (vs ~50 for plain Ornith). Use a plain GGUF if you need the model's full 256k.

How it was made

Ornith-1.0-35B is an architectural sibling of Qwen3.6-35B-A3B: identical tensor set, identical dims (40 blocks, 256 experts, 2048 embd), identical 248320-token vocabulary. The 20 MTP tensors (blk.40.* — one extra transformer layer plus the nextn projection/norms) from unsloth's Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf were appended to ornith-1.0-35b-Q4_K_M.gguf, with qwen35moe.block_count patched 40→41 and qwen35moe.nextn_predict_layers=1 added:

import gguf
from gguf import GGUFReader, GGUFWriter, GGUFValueType

orn = GGUFReader("ornith-1.0-35b-Q4_K_M.gguf")
mtp = GGUFReader("Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf")
writer = GGUFWriter("ornith-1.0-35b-MTP-graft-Q4_K_M.gguf", "qwen35moe", use_temp_file=False)

for field in orn.fields.values():
    if field.name == gguf.Keys.General.ARCHITECTURE or field.name.startswith("GGUF."):
        continue
    val_type = field.types[0]
    sub_type = field.types[-1] if val_type == GGUFValueType.ARRAY else None
    val = field.contents()
    if field.name == "qwen35moe.block_count":
        val = 41
    writer.add_key_value(field.name, val, val_type, sub_type=sub_type)
writer.add_key_value("qwen35moe.nextn_predict_layers", 1, GGUFValueType.UINT32)

mtp_extra = [t for t in mtp.tensors if t.name.startswith("blk.40.")]
all_tensors = list(orn.tensors) + mtp_extra
for t in all_tensors:
    writer.add_tensor_info(t.name, t.data.shape, t.data.dtype, t.data.nbytes, t.tensor_type)

writer.write_header_to_file()
writer.write_kv_data_to_file()
writer.write_ti_data_to_file()
for t in all_tensors:
    writer.write_tensor_data(t.data, tensor_endianess=orn.endianess)
writer.close()

The same recipe works for any qwen35moe-architecture fine-tune (abliterated variants, etc.).

Credits

• DeepReinforce — Ornith-1.0-35B base model (MIT)
• Qwen / unsloth — Qwen3.6-35B-A3B MTP head (Apache 2.0)
• skinnyctax and wang-yang — pioneered the Frankenstein-MTP graft approach at Q6_K/Q8_0

Base model weights are unchanged; this is purely a head graft plus metadata patch. MIT license inherited from the base model; the grafted head tensors remain Apache 2.0.