document-classification-v1 — open-weight

An open-weight, open-vocabulary document classifier you can download and run. Supply any set of text labels at inference; the model scores a document image against them by calibrated cosine and returns a per-label match probability. No fixed class list, no per-class training.

The open-weight sibling of the commercial flagship document-classification-v2. It ships as two self-contained ONNX graphs — an image tower and a text tower — that you run with onnxruntime. embed_dim: 1024; classification p = sigmoid(scale·cos + bias) (calibration in modules/omni-image/config.json).

• 🎯 Try it: document-classification-demo
• 🏆 Leaderboard: document-classification-leaderboard
• 📊 Benchmark: document-classification-benchmark
• 🏵️ Flagship (commercial): document-classification-v2

Results (macro-F1, zero-shot)

Benchmark	v1 (open)	v2 (commercial)	best cloud VLM
DocLayNet	0.89	0.88	0.83
Forms	0.80	1.00	1.00
Tobacco	0.62	0.69	0.85
OOD (unseen types)	0.87	0.97	—
OOV (synonym wording)	0.74	0.80	—

Every entry is scored by the same open scorer — full ranking, plus a generalist zero-shot baseline and each cloud model, on the leaderboard. v1 leads on the visual document-type track (DocLayNet) as a free download; like all embedding models it trails large VLMs on Tobacco (a read-the-header task). ~3–7 docs/s on an A40 (image branch).

Usage (ONNX)

import numpy as np, onnxruntime as ort, json
from transformers import AutoProcessor, AutoTokenizer
from huggingface_hub import hf_hub_download

R = "nutrientdocs/document-classification-v1"
img_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/image_model.onnx"))
txt_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/text_model.onnx"))
cal = json.load(open(hf_hub_download(R, "modules/omni-image/config.json")))["calibration"]
proc = AutoProcessor.from_pretrained(R, subfolder="modules/omni-image")   # bundled preprocessor
tok  = AutoTokenizer.from_pretrained(R, subfolder="modules/omni-image")   # bundled tokenizer (right-pad + attention_mask)

from PIL import Image
labels = ["invoice", "letter", "memo", "form", "scientific article", "resume"]
pix = proc(images=[Image.open("doc.png").convert("RGB")], return_tensors="np")["pixel_values"].astype(np.float16)
ie = img_sess.run(["image_emb"], {"pixel_values": pix})[0]                      # [1, 1024] L2
enc = tok(labels, padding=True, truncation=True, max_length=64, return_tensors="np")
te = txt_sess.run(["text_emb"], {"input_ids": enc["input_ids"].astype(np.int64),
                                 "attention_mask": enc["attention_mask"].astype(np.int64)})[0]  # [N,1024] L2
cos = (ie @ te.T)[0]
probs = 1 / (1 + np.exp(-(cal["scale"] * cos + cal["bias"])))
print(dict(zip(labels, probs.round(3).tolist())))

What's in this repo

• modules/omni-image/{image_model.onnx, text_model.onnx} — the image + text towers (fp16, onnxruntime).
• modules/omni-image/{config.json, preprocessor_config.json, tokenizer.json} — calibration + the preprocessor and tokenizer needed to run them. That's it — nothing else required.

Open weights under Apache-2.0 — free to download and run. For the higher-accuracy commercial flagship (on-prem, calibrated), see document-classification-v2.

About the author

This project is maintained and funded by Nutrient - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.