| --- |
| license: other |
| license_name: nutrient-commercial |
| pipeline_tag: zero-shot-image-classification |
| language: |
| - en |
| tags: |
| - zero-shot-image-classification |
| - image-classification |
| - document-ai |
| - document-image-classification |
| - open-vocabulary |
| datasets: |
| - nutrientdocs/document-classification-benchmark |
| metrics: |
| - accuracy |
| --- |
| |
| # document-classification-v2 · _commercial_ |
|
|
| **Classify any document image against labels you choose at runtime.** `document-classification-v2` is an |
| open-vocabulary, zero-shot document classifier: supply one label + a short description per class at |
| inference, and it scores a document page against *any* such set — invoices, letters, memos, forms, |
| scientific articles, resumes, and whatever label you type next. No fixed class list, no per-class training. |
|
|
| It is the document specialist in a field of generalists. General open-vocab models were trained on web |
| photos; this one is built for document pages — and it shows on the |
| [leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard). **Weights are |
| commercial** (not downloadable here); this page is a spec + scorecard. |
|
|
| - 🎯 **Try it:** [document-classification-demo](https://huggingface.co/spaces/nutrientdocs/document-classification-demo) |
| - 🏆 **Leaderboard:** [document-classification-leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard) |
| - 📊 **Benchmark:** [document-classification-benchmark](https://huggingface.co/datasets/nutrientdocs/document-classification-benchmark) |
|
|
| ## Results |
|
|
| Macro-F1, zero-shot, on the held-out benchmark — higher is better. The flagship vs the previously-shipped |
| in-house model and a cloud frontier VLM reference. |
|
|
| <!-- RESULTS-TABLE:START — filled from leaderboard/leaderboard.json by the scoring run --> |
| | Benchmark (macro-F1) | `document-classification-v2` | best generalist | best cloud VLM | |
| | --- | ---: | ---: | ---: | |
| | **DocLayNet** (page types) | **0.88** | 0.63 | 0.83 | |
| | **Forms** | **1.00** | 0.11 | 1.00 | |
| | **Tobacco** (doc types) | 0.69 | 0.43 | **0.85** | |
| | **OOD** (unseen doc types) | **0.97** | — | — | |
| | **OOV** (synonym wording) | **0.80** | — | — | |
|
|
| Latency: **~8 docs/s on an A40** (p50 106 ms / p95 194 ms). The open-weight |
| [`v1`](https://huggingface.co/nutrientdocs/document-classification-v1) runs ~3–7 docs/s on the same GPU. |
| <!-- RESULTS-TABLE:END --> |
|
|
| Every model — ours and cloud — is scored by the same open macro-F1 scorer; full per-model ranking on the |
| [leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard). |
|
|
| - **Matches/leads the cloud on visual document-type tracks** (DocLayNet, Forms) at **zero per-request API |
| cost** — and runs on a single GPU. |
| - **Trails on Tobacco** (0.69 vs 0.85): that track rewards *reading* fine header text (memo vs letter vs |
| email) — a large VLM reads it; an embedding model can't. |
| - **OOD robustness:** on document types absent from training (invoices, handwriting, charts, tables), 0.97 |
| macro-F1. **Open-vocab (OOV):** under never-seen synonym label wording, 0.80 — matches the *concept*, not |
| your exact string. (Cloud VLMs train on ~all data, so these held-out axes aren't reported for them.) |
|
|
| ## Intended use & limits |
|
|
| - **Use it for:** zero-shot classification of document page images in a free-label setting — the caller |
| supplies the candidate labels (and optional descriptions). Multi-page documents supported; optional page |
| OCR sharpens fine-grained form / tax-code distinctions. |
| - **Limits:** optimized for **document** imagery; English label strings are the primary target. Scores are |
| per-label match probabilities (independent per label), not a softmax across the set. |
|
|
| ## License & data |
|
|
| The model **weights** are offered under a commercial Nutrient license — deployed on-prem, so your documents |
| never leave your infrastructure. The training set is not redistributed. Evaluation runs on the held-out |
| [document-classification-benchmark](https://huggingface.co/datasets/nutrientdocs/document-classification-benchmark). |
|
|
| > ### 📩 Get access |
| > |
| > `document-classification-v2` is commercial and its weights are not downloadable here. To run it on-prem — |
| > open-vocabulary, calibrated, private — **contact Nutrient: |
| > [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).** |
|
|
| ## About the author |
|
|
| <a href="https://nutrient.io/"> |
| <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" /> |
| </a> |
|
|
| This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks. |
|
|
