dsaad68/LFM2.5-Embedding-350M-ONNX-int8

LFM2.5-Embedding-350M — ONNX (int8)

int8 ONNX export of LiquidAI/LFM2.5-Embedding-350M, a 354M-parameter bidirectional LFM2 embedding model (hybrid short-conv + attention, CLS pooling, 1024-dimensional, query: / document: prompt prefixes). Weights are dynamically quantized to int8 (onnxruntime.quantization.quantize_dynamic) from an fp32 base — ~356 MB, half the size of the fp16 variant.

Accuracy is CPU-dependent: ONNX Runtime's dynamic-quant integer kernels differ by architecture — measured min cosine vs the PyTorch reference ~0.989 on Apple Silicon (ARM) but ~0.965 on x86-64 (AVX2). Retrieval ranking was preserved on both. If you need tight parity on x86 (or GPU speed), use the fp16 variant — on WebGPU int8 runs ~5× slower because its integer ops aren't GPU-accelerated.

Files

File	Notes
onnx/model.onnx	int8 dynamic-weight-quant graph, single self-contained file (~356 MB)

The graph is a single model with two outputs (one weight copy serves both):

Output	Shape	Use
sentence_embedding	(batch, 1024)	CLS pooling + L2 normalization baked in — use this for retrieval
last_hidden_state	(batch, sequence, 1024)	raw token embeddings — pool externally (used by the sentence-transformers ONNX backend)

Usage

Prompt prefixes are required (asymmetric retrieval model): prepend query: to queries and document: to passages. Max sequence length is 512 tokens. Outputs are fp32.

sentence-transformers (ONNX backend)

Requires optimum[onnxruntime]; onnx/model.onnx is found automatically:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("dsaad68/LFM2.5-Embedding-350M-ONNX-int8", backend="onnx", trust_remote_code=True)
q = model.encode(["How do I reset my password?"], prompt_name="query",
                 normalize_embeddings=True)
d = model.encode(["Click 'Forgot password' on the sign-in page."],
                 prompt_name="document", normalize_embeddings=True)
print(q @ d.T)

ONNX Runtime (Python)

import numpy as np, onnxruntime as ort
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

repo = snapshot_download("dsaad68/LFM2.5-Embedding-350M-ONNX-int8")
tok = AutoTokenizer.from_pretrained(repo)
sess = ort.InferenceSession(f"{repo}/onnx/model.onnx")

texts = ["query: How do I reset my password?",
         "document: Click 'Forgot password' on the sign-in page."]
enc = tok(texts, padding=True, truncation=True, max_length=512, return_tensors="np")
emb = sess.run(["sentence_embedding"],
               {"input_ids": enc["input_ids"].astype(np.int64),
                "attention_mask": enc["attention_mask"].astype(np.int64)})[0]
# emb: (2, 1024) fp32 unit vectors; cosine similarity = dot product
print(emb @ emb.T)

Browser (onnxruntime-web)

Runs under the WASM execution provider (CPU-in-browser); a single self-contained file, no external-data sidecar. Prefer the fp16 variant for WebGPU. See the web-test harness for a worked example.

Export details

• fp32 base exported with torch.onnx.export (opset 17, dynamic batch + sequence axes) from the upstream remote-code Lfm2BidirectionalModel with attn_implementation="eager", then dynamically quantized to int8 weights.
• Parity gate (run in CI on every publish, on ARM): per-vector cosine vs SentenceTransformer(..., trust_remote_code=True) ≥ 0.985 on both outputs, plus retrieval-ranking equality.
• Why fp32 / bf16 / fp8 / int4 were not shipped: learning notes.

License

The model weights are redistributed under the LFM Open License v1.0, unchanged from the original LiquidAI/LFM2.5-Embedding-350M release. All credit for the model itself goes to Liquid AI; this repository only converts it to ONNX.