NicoLab28/ClipProj-MiniMax-H3

ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B

Projection matrices that let a Qwen3-VL-4B replace the Qwen3-VL-32B text encoder of MiniMax H3.

15.7 GB → 5.2 GB of VRAM, with no change to the diffusion model, the VAEs or the sampler.

⚠️ Proof of concept — working, but a proof of concept. It runs and produces good video, and every number below was measured on real hardware. Built and tested on a single setup (Windows 11, NVIDIA, ComfyUI 0.31.0) with deliberately limited exploration.

These files are useless on their own. They require the custom node: github.com/nicolab28/ComfyUI-ClipProj

Now in safetensors

Every matrix is available as .safetensors. Prefer it. A .pt goes through pickle, which can execute arbitrary code the moment it is opened — an absurd risk for files that hold nothing but tensors and a handful of scalars. The scalars now live in the safetensors header and the node converts them back on load.

Reading them needs node v0.1.2. The .pt files are kept alongside for now so that anyone still on an older node has something that works; they will go away in a later release.

Requested by @lucas-ai26, who was right.

Re-download if you got the first release

The first release was missing the attention-sink vector, and short prompts suffered for it. The first token of a sequence is an attention sink: its direction is constant from one prompt to the next (cosine 1.0000 measured over 1966 prompts) and it carries nothing from the text, yet its norm reaches 16 500 against 291 for a text token. Calibration excluded it, rightly — its extreme values would wreck the statistics — but the node projected it anyway, through a matrix that had never seen one, producing an arbitrary vector of enormous norm. Invisible on a 200-token prompt where it is 0.5 % of the positions, ruinous on a 7-token one where it is 14 %.

Every matrix here now carries the measured sink vector and substitutes it at inference. Since the vector is constant, this is not an approximation: it is the value the 32B would have produced.

Also new in this release: an 8B matrix, an int8_convrot matrix, and ref2va now works — see below.

What this is

MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a [seq, 5120] tensor. This repository provides a learned linear map that lets a much smaller Qwen3-VL produce the same conditioning:

cond = ((h - mean_in) / std_in) @ W * std_out + mean_out

It works because every Qwen3-VL shares the same tokenizer (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. Calibration is plain ridge regression — no gradients, no epochs, no learning rate.

Files

Put them in ComfyUI/models/clip_projections/.

Start with h3_qwen3vl_4b_tap24. It is calibrated on eight times more data than anything else here.

File	Use it for	Encoder	Corpus	Test cosine
h3_qwen3vl_4b_tap24	the default	any Qwen3-VL-4B	2 000 prompts	0.712
h3_qwen3vl_8b_tap24	any 8B encoder	any Qwen3-VL-8B	203 prompts	0.731
h3_control_identity	control — run it once	—	—	—
h3_control_zero	control — run it once	—	—	—
h3_qwen3vl_4b_CONDPROJ_tap24	see below	any Qwen3-VL-4B	240 prompts	0.845 *
h3_qwen3vl_8b_CONDPROJ_tap24	see below	any Qwen3-VL-8B	203 prompts	0.860 *
h3_qwen3vl_4b_int8convrot_tap24	comparison only	any Qwen3-VL-4B	240 prompts	0.697
h3_qwen3vl_4b_tap24_200prompts	comparison only	any Qwen3-VL-4B	200 prompts	0.699

* measured in a different space and not comparable to the other rows. See below, and do not read it as better.

Every matrix works on any variant of its own size: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on. The 8B matrices need an 8B encoder though, 4096 input dimensions instead of 2560, and the node checks and refuses a mismatch.

The 8B matrices need an 8B encoder — different input width, 4096 instead of 2560, so they are not interchangeable with the 4B ones. The node checks the width and refuses a mismatch.

Every matrix here works on any variant of its own size: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on.

The CONDPROJ matrices, and why they are not the default

They are kept because the comparison posted on r/StableDiffusion ran on them, so the links have to keep working. They are not better, and the story is worth telling because the mistake was instructive.

The DiT does not consume the conditioning as it arrives: it first passes it through condition_proj, a Linear(5120 → 5376) feeding the token refiner. That layer's spectrum is very uneven, a factor of 45 between the top and bottom deciles of its singular values, and 52 % of the energy in 10 % of the directions. Plain ridge regression ignores this and spends as much effort on a direction the DiT will multiply by 0.10 as on one it will multiply by 37. Calibrating against the output of that layer instead, then mapping back through the pseudo-inverse, should therefore minimise the error the DiT actually sees. The cosine went from 0.697 to 0.845 on the 4B and 0.731 to 0.860 on the 8B.

Then I compared what the two matrices actually output, on random inputs:

4B   CONDPROJ against unweighted, same corpus     cosine 0.999998
8B   CONDPROJ against unweighted, same corpus     cosine 0.999999
4B   CONDPROJ against the 2000-prompt matrix      cosine 0.744034

They are the same function. The entire gain was an artefact of measuring in a different space: applying an invertible transform to both vectors before taking a cosine flatters agreement on the dominant directions.

The reason is simple in hindsight. Unregularised least squares is invariant to an invertible linear transform of the targets, so fitting in one space and mapping back recovers the same map. Only the ridge penalty breaks that invariance, and with 37 851 training tokens against λ = 1000 it barely binds. The idea would matter with far less data or far stronger regularisation. It does not matter here.

The idea came from u/stddealer on r/StableDiffusion, and it was a good one. The measurement is on me: I published the cosine before checking whether the matrix had changed at all.

Measured results

Corpus	Tokens	Cross-prompt CKA	Test cosine	Test R²
200 prompts	37 361	0.95	0.699	0.490
2 000 prompts	288 608	0.92	0.712	0.507

Eight times the data buys 1.8 % of cosine: the linear projection is at its ceiling, not starved of data. Going further needs an MLP, not more prompts.

A cosine of 0.71 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts (four distinct cuts with no bleed between them), fl2va with first and last frame, and ref2va with a reference image.

Fidelity does not collapse on short prompts, contrary to what one might expect from a corpus of long ones: measured per-token cosine goes from 0.937 at 80 words down to 0.908 at 2 words, a 3 % decline, once the sink is handled.

Run the controls first

The two control matrices exist to prove the learned matrix is doing the work rather than the diffusion model. Same prompt, same seed, only the matrix changes:

Matrix	Output for "a red ball on a wood table"
h3_control_zero.safetensors	a countryside landscape — the prompt is entirely ignored
h3_control_identity.safetensors	a golden object in flames — unusable
h3_qwen3vl_4b_tap24.safetensors	the red ball on a wood table

‖W_identity‖ = 50.6 against ‖W_learned‖ = 52.4 — near-identical energy, so the difference is structural, not a matter of scale.

If the identity control ever looks fine, the learned matrix adds nothing — and you want to know that before trusting it.

Known limitations

Named references depend on the encoder, not on the projection. This was measured rather than guessed: asking the encoder to describe a person in plain text — which bypasses the matrix entirely — shows where the knowledge actually stops. The 4B places Scarlett Johansson correctly as Black Widow but believes she has dark brown hair; the 8B describes her correctly as blonde with blue eyes. So when a proper noun renders as the wrong person, question the encoder first.

The practical workaround is to describe rather than name: "the actress X as [role], blonde, ..." recovers an identity that the bare name loses, on both the 4B and the 8B. A name is a fragile signal carried by two or three tokens; a description spreads it over a dozen redundant ones and the reconstruction error averages out instead of accumulating.

Note that this does not apply to ref2va: there the identity comes from the reference image, so the encoder's factual gaps stop mattering.

Quantisation costs facts. The same test on int8_convrot against bf16 shows two factual errors appearing under quantisation. Fine for general use, worth knowing if your prompts lean on proper nouns.

Speech in languages other than English degrades. Reported and reproduced with French: the 32B pronounces it cleanly, a projected 4B or 8B does not. This is not a corpus problem — measured on identical English prompts differing only in the quoted line, French tokens reconstruct at 0.8974 against 0.8996 for English, which is noise. A cosine of 0.90 is ample for visual semantics and insufficient for phonetics; the DiT's audio branch is far more demanding than its image branch, and a language the model handles less confidently has less margin to absorb the error.

Required models

Role	Model
Diffusion model + VAEs	Comfy-Org/MiniMax-H3
Text encoder, 4B	Comfy-Org/Krea-2 → text_encoders/qwen3vl_4b_fp8_scaled.safetensors
Text encoder, 8B	any ComfyUI-format Qwen3-VL-8B (the 8B matrix expects 4096 input dims)

The 32B text encoder is no longer needed — that is the entire point.

Licence and responsibility

These matrices are released under MIT, like the node.

They are derived from the activations of both models, and their legal status is unclear. They are provided as-is, for research, with no claim of ownership over anything derived from the underlying models.

• Qwen3-VL is published by Alibaba under Apache 2.0. Read and comply with its terms and acceptable-use policy.
• MiniMax H3 ships under a custom licence. Read it before any use, particularly commercial.

This project is not affiliated with, endorsed by, or connected to Alibaba / Qwen, MiniMax, or Comfy Org.

You remain responsible for what you generate and for complying with the licences of every model you load.

Credits

Vibe-coded with Anthropic Claude Code (Opus 5). Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected.