license: other
license_name: nutrient-commercial
pipeline_tag: zero-shot-image-classification
language:
- en
tags:
- zero-shot-image-classification
- image-classification
- document-ai
- document-image-classification
- open-vocabulary
datasets:
- nutrientdocs/document-classification-benchmark
metrics:
- accuracy
document-classification-v2 · commercial
Classify any document image against labels you choose at runtime. document-classification-v2 is an
open-vocabulary, zero-shot document classifier: supply one label + a short description per class at
inference, and it scores a document page against any such set — invoices, letters, memos, forms,
scientific articles, resumes, and whatever label you type next. No fixed class list, no per-class training.
It is the document specialist in a field of generalists. General open-vocab models were trained on web photos; this one is built for document pages — and it shows on the leaderboard. Weights are commercial (not downloadable here); this page is a spec + scorecard.
- 🎯 Try it: document-classification-demo
- 🏆 Leaderboard: document-classification-leaderboard
- 📊 Benchmark: document-classification-benchmark
Results
Macro-F1, zero-shot, on the held-out benchmark — higher is better. The flagship vs the previously-shipped in-house model and a cloud frontier VLM reference.
| Benchmark (macro-F1) | document-classification-v2 |
best generalist | best cloud VLM |
|---|---|---|---|
| DocLayNet (page types) | 0.88 | 0.63 | 0.83 |
| Forms | 1.00 | 0.11 | 1.00 |
| Tobacco (doc types) | 0.69 | 0.43 | 0.85 |
| OOD (unseen doc types) | 0.97 | — | — |
| OOV (synonym wording) | 0.80 | — | — |
Latency: ~8 docs/s on an A40 (p50 106 ms / p95 194 ms). The open-weight
v1 runs ~3–7 docs/s on the same GPU.
Every model — ours and cloud — is scored by the same open macro-F1 scorer; full per-model ranking on the leaderboard.
- Matches/leads the cloud on visual document-type tracks (DocLayNet, Forms) at zero per-request API cost — and runs on a single GPU.
- Trails on Tobacco (0.69 vs 0.85): that track rewards reading fine header text (memo vs letter vs email) — a large VLM reads it; an embedding model can't.
- OOD robustness: on document types absent from training (invoices, handwriting, charts, tables), 0.97 macro-F1. Open-vocab (OOV): under never-seen synonym label wording, 0.80 — matches the concept, not your exact string. (Cloud VLMs train on ~all data, so these held-out axes aren't reported for them.)
Intended use & limits
- Use it for: zero-shot classification of document page images in a free-label setting — the caller supplies the candidate labels (and optional descriptions). Multi-page documents supported; optional page OCR sharpens fine-grained form / tax-code distinctions.
- Limits: optimized for document imagery; English label strings are the primary target. Scores are per-label match probabilities (independent per label), not a softmax across the set.
License & data
The model weights are offered under a commercial Nutrient license — deployed on-prem, so your documents never leave your infrastructure. The training set is not redistributed. Evaluation runs on the held-out document-classification-benchmark.
📩 Get access
document-classification-v2is commercial and its weights are not downloadable here. To run it on-prem — open-vocabulary, calibrated, private — contact Nutrient: nutrient.io/contact-sales.
About the author
This project is maintained and funded by Nutrient - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.