mlboydaisuke/CLIP-ViT-B32-ExecuTorch

clip_vit_b32_image — ExecuTorch

• Source: openai/clip-vit-base-patch32
• License: MIT
• Input: [[1, 3, 224, 224]] — RGB, CLIP norm (mean .481/.458/.408, std .269/.261/.276), 224x224
• Output: image embedding [1,512] (unnormalized; L2-normalize before cosine)

Variants

All variants take and return fp32 tensors — swap the .pte file, keep your app code.

build	file	size (MB)	parity vs fp32 eager (worst corr)	Mac median (ms)*
fp32	clip_vit_b32_image_xnnpack_fp32.pte	351.6	1.000000	19.0
fp16	clip_vit_b32_image_xnnpack_fp16.pte	180.7	0.999996	26.1
Core ML (fp16, iOS)	clip_vit_b32_image_coreml_all.pte	176.2	0.999998	3.5

The Core ML build is the same graph lowered to Apple's Neural Engine instead of XNNPACK, which is CPU-only. Measured on an iPhone 17 Pro across seven models, it runs 3.5x to 13.9x faster (median 12x) at roughly half the file size — for example Depth-Anything-V2-Small at 500.8 ms against 42.7 ms, and MODNet at 81.7 ms against 5.9 ms. It computes in fp16 and is iOS-only; the XNNPACK files stay the portable option and are what runs on Android.

*Mac arm64, single process, median of 10 — a reference point for relative cost only, not a device number (torch eager fp32 on the same machine: 18.5 ms).

Checked in the task's own units

Correlation is a first filter, and on an embedding it is a weak one: a build can read 0.999 against eager and still move an answer, because what decides is whether the error is smaller than the gap between the answer a query gets and the runner-up.

The test is near-duplicate retrieval over 52 photographs, each also present centre-cropped to 90% and re-encoded as JPEG. Every item has one obviously correct nearest neighbour, and each build has to find it.

build	correct neighbour kept	worst score shift	budget spent
fp32	104 of 104	0.0000	0%
fp16	104 of 104	0.0023	32%
Core ML (fp16, iOS)	104 of 104	0.0030	43%

The closest decision eager makes on this set is 0.0071, and the bar is half of it. An earlier version of this card cleared int8 on "cosine similarity of the embeddings", which is correlation wearing a task metric's clothes — it never asked whether a retrieval answer moved.

Withdrawn: int8 (2026-08-27)

clip_vit_b32_image_xnnpack_int8.pte was published and has been withdrawn. It was cleared on "cosine similarity of the image embeddings, median 0.9988 over 10 real images", which never asked whether a retrieval answer moved.

Re-measured on the test above it still matches all 104 copies to their originals, but it moves a pair score by 0.0404 against a closest decision of 0.0071 — 567% of the room available, against a bar of 50%. Its zero-shot label also moved on one image of the 104, where no other build moved any. And the number is generous: this shelf calibrated the int8 build on these same photographs.

At 95.9 MB it was the smallest build and size was the reason to reach for it. fp16 is 180.7 MB and spends 32% of the same budget; the Core ML build is 176.2 MB, spends 43%, and runs 5x faster.

Verification (executorch 1.4.0, torch 2.13.0)

Parity is measured against the fp32 eager model on real image input; corr is the correlation over all elements of each output tensor.

output	shape	max_abs_diff	corr
0	[1, 512]	9.179e-06	1.000000

XNNPACK delegate coverage (fp32): 69.3% (390/563 ops); ops left on the portable kernels: aten.expand_copy.default x49, aten.native_layer_norm.default x26, aten.mul.Scalar x24, aten.logical_not.default x24, aten.eq.Scalar x12, aten.full_like.default x12, aten.any.dim x12, aten.where.self x12, aten.embedding.default x1, aten.select_copy.int x1

Conversion

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models)

This repo holds both towers: clip_vit_b32_image_xnnpack_fp32.pte (image) and clip_vit_b32_text_xnnpack_fp32.pte (text, fixed len 77 + attention mask). L2-normalize both embeddings, then cosine-match.