FINAL-Bench/ZTC-Judge-4B
Text Classification β’ 5B β’ Updated β’ 57 β’ 31
chohyunkyu
chohyun
AI & ML interests
None yet
Recent Activity
liked a model about 16 hours ago
FINAL-Bench/ZTC-Judge-4B liked a model about 16 hours ago
FINAL-Bench/ZTC-Judge-9B liked a model about 16 hours ago
FINAL-Bench/ZTC-Judge-27BOrganizations
None yet
liked 3
models about 16 hours ago
upvoted 2
collections about 16 hours ago
liked a
Space about 16 hours ago
liked 2
Spaces 1 day ago
reacted to mayafree's post with β€οΈ 1 day ago
Post
1971
JEV Ecosystems β every answer-verification vendor publishes a benchmark, and every one of them wins it. So we ran 13 of them on one test set: 2,018 items, identical labels, same grading code.
π― Leaderboard
mayafree/typed-decision-leaderboard
π Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
π§ͺ Try it β ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1οΈβ£ Only three systems clear 0.70 β ZTC (397B) 0.7364 Β· JEV 0.7350 Β· ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2οΈβ£ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3οΈβ£ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 β a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp Β· JEV β0.07 pp Β· random β0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it β paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
π― Leaderboard
mayafree/typed-decision-leaderboard
π Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
π§ͺ Try it β ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1οΈβ£ Only three systems clear 0.70 β ZTC (397B) 0.7364 Β· JEV 0.7350 Β· ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2οΈβ£ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3οΈβ£ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 β a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp Β· JEV β0.07 pp Β· random β0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it β paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
reacted to ginigen-ai's post with π₯ 2 days ago
Post
4420
A local edge VLM you can run on a phone β with a calibration readout attached.
ginigen-ai/Edge-4B-TELL
Image in, answer out, nothing leaving the device. Google's Gemma 4 E4B QAT checkpoint carried unmodified, with the vision and audio projector, plus one thing that is ours: GINIGEN TELL, a 10 KB readout that estimates whether the answer it just gave is likely to be wrong.
On a Galaxy S25: zero network calls, 3.6 GB resident, a 12.6 MB inference binary.
Calibration matters more here than on a server: nothing downstream catches a bad answer. No retrieval, no second opinion, no reviewer. The model is alone with the user.
And its own confidence is unusable. Prompted for it, this checkpoint averages 0.863 over 665 Korean disaster-procedure questions β ranking answers by it gives AUROC 0.441, below a coin flip. It sounds more certain when it is wrong.
TELL reads the last-layer hidden state instead of asking. Same questions, 0.759. Surface cues (length, formatting) already reach 0.736, so the readout clears that baseline by +0.023 Β± 0.009 (2.6Ο). We publish the baseline because without it, "the hidden state carries the signal" is unfalsifiable.
Same job as JEV: a confidence number you can act on instead of the model's own. Different structure, and on a device that splits three ways.
No second model β JEV is a separate judge reading the answer as text; we fill that slot with a 10 KB vector.
Zero generated tokens β a judge writes its verdict, TELL re-reads a finished computation (3.8 s on an S25).
No network β a verdict fetched over an API stops when the signal does.
The trade is real: a readout is fitted per checkpoint, so on a server the judge wins. On a phone there is no second model to run.
TELL never says what the right answer is. It says whether the answer wobbled, and a low score falls back to source text bundled with the app. Shipping today in HeliGO, an offline disaster-response app.
ginigen-ai/Edge-4B-TELL
Image in, answer out, nothing leaving the device. Google's Gemma 4 E4B QAT checkpoint carried unmodified, with the vision and audio projector, plus one thing that is ours: GINIGEN TELL, a 10 KB readout that estimates whether the answer it just gave is likely to be wrong.
On a Galaxy S25: zero network calls, 3.6 GB resident, a 12.6 MB inference binary.
Calibration matters more here than on a server: nothing downstream catches a bad answer. No retrieval, no second opinion, no reviewer. The model is alone with the user.
And its own confidence is unusable. Prompted for it, this checkpoint averages 0.863 over 665 Korean disaster-procedure questions β ranking answers by it gives AUROC 0.441, below a coin flip. It sounds more certain when it is wrong.
TELL reads the last-layer hidden state instead of asking. Same questions, 0.759. Surface cues (length, formatting) already reach 0.736, so the readout clears that baseline by +0.023 Β± 0.009 (2.6Ο). We publish the baseline because without it, "the hidden state carries the signal" is unfalsifiable.
Same job as JEV: a confidence number you can act on instead of the model's own. Different structure, and on a device that splits three ways.
No second model β JEV is a separate judge reading the answer as text; we fill that slot with a 10 KB vector.
Zero generated tokens β a judge writes its verdict, TELL re-reads a finished computation (3.8 s on an S25).
No network β a verdict fetched over an API stops when the signal does.
The trade is real: a readout is fitted per checkpoint, so on a server the judge wins. On a phone there is no second model to run.
TELL never says what the right answer is. It says whether the answer wobbled, and a low score falls back to source text bundled with the app. Shipping today in HeliGO, an offline disaster-response app.
liked a
model 2 days ago
liked a
Space 5 days ago
reacted to ginigen-ai's post with π€ 5 days ago
Post
2933
OpenRouter Leaderboard β every model, every provider, one comparable table. Price, precision, uptime, measured latency and language quality on the same axes.
Building it turned up three things.
We graded 330 models on Korean and two axes collapsed.
Honorifics β only 8.5% earn an A
Knowledge of Korean institutions β 9.4%
Every other axis sits above 31%
Fluency hides it. A model can write clean, natural Korean and still attach an honorific to a coffee cup. Fluent and wrong at the same time is worse than obviously broken, because nobody catches it in review.
A 2023 model beats the 2026 flagships. gpt-3.5-turbo-16k scores a perfect 3.00. Korean cannot be inferred from release date, parameter count or English benchmarks β it has to be measured, per model.
Quality, value and speed are three different models. Across five axes, the same model almost never takes two columns.
425 models, latency measured on 329 on a paid API, Korean graded on 330. Three languages, three currencies, daily refresh, open API, no key.
π https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard π ginigen-ai/open-router-leaderboard
Building it turned up three things.
We graded 330 models on Korean and two axes collapsed.
Honorifics β only 8.5% earn an A
Knowledge of Korean institutions β 9.4%
Every other axis sits above 31%
Fluency hides it. A model can write clean, natural Korean and still attach an honorific to a coffee cup. Fluent and wrong at the same time is worse than obviously broken, because nobody catches it in review.
A 2023 model beats the 2026 flagships. gpt-3.5-turbo-16k scores a perfect 3.00. Korean cannot be inferred from release date, parameter count or English benchmarks β it has to be measured, per model.
Quality, value and speed are three different models. Across five axes, the same model almost never takes two columns.
425 models, latency measured on 329 on a paid API, Korean graded on 330. Three languages, three currencies, daily refresh, open API, no key.
π https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard π ginigen-ai/open-router-leaderboard
liked 3
models 7 days ago
upvoted a collection 7 days ago
liked a
Space 7 days ago
liked 2
Spaces 16 days ago