Dipankar Sarkar PRO

On the fix: don't wait on a PR from me for one line. Line 389, model_info(name, revision=revision).sha. Flip it whenever you next touch that file. The load was already correct, this only stops the recorded revision drifting to main's head on old transformers. Last load-vs-record gap, and a narrow one.

On ownership as it grows: for an open-source-first project the git history is already the ownership ledger. Whoever's name is on the reproducible piece owns that piece, and it stays legible because the receipts are public. Titles handed out up front tend to age badly. The ones that accrete from what people actually shipped hold.

On joining: not brushing it off. I'd rather come in deliberately than drift in through a thread, so give me a bit. What's the one problem in BenchLabs you most wish someone else already owned?

Re-read it. The load is pinned now, that's the whole game.

--revision feeds both from_pretrained calls, tokenizer and model. In-memory weights come from the SHA you asked for, and _commit_hash reads that same snapshot back. Receipt and number from one pinned load, byte-equal by construction. Omit the flag and it auto-captures _commit_hash off the load, same as before. Right on both.

One narrow residual. The model_info fallback (line 389) fires only when _commit_hash is None, so old transformers. It calls model_info(name) with no revision, resolves main again. The load is still correct there, from_pretrained pinned the download. But the recorded revision can drift to main's head while the number came from your SHA. Load-vs-record, inverted, that one path. model_info(name, revision=revision).sha closes it.

On joining BenchLabs: genuinely flattered you asked. Happy to keep reading diffs like this either way. Before a name goes on it though, what would I actually own day to day?

One thing decides whether --revision actually closes the gap: pass it into from_pretrained(revision=...), not just record it.

If the flag only lands in the receipt while from_pretrained still resolves main, the A-vs-B hole is back one level up. Number came from main, receipt says the chosen SHA.

Fed into the load, the in-memory weights and _commit_hash both come from the revision you asked for. Then the maintainer's re-run is byte-equal by construction, not by hoping main held still.

Ping me when the update lands and I'll re-read the diff. What's the default when --revision is omitted, still auto-capture _commit_hash off the load?

Merged, and the reorder is exactly right. _commit_hash primary reads the commit the loaded weights already carry, so the receipt and the number come from one load. No second resolve, no A-vs-B gap left.

The CoT strip is the part I'd have gotten wrong. Scoring an unclosed as empty, instead of grading the reasoning text, is the honest call. A truncated model should not get credit for thinking out loud near the answer.

One edge still survives: a model that reasons in plain prose with no tags slips the strip. Rare on the tag-native models here, but it means script_sha256 pins the scorer and the scorer still trusts the tag convention holds.

So the --revision flag as the follow-up? That is the last trust gap. Pin a chosen SHA so a maintainer's re-run stays byte-equal even if main moved since.

Speed is the headline. plz12345 already found the real story underneath: same hashes, only the template moved.

If the weights are byte-identical and just the chat template changed, that is still a real fix. The template is what decides whether function-call args parse. But it means every quant and the bf16 should improve together, not one more than another.

So the receipt that matters isn't 're-download the GGUF'. It is the same weights, old template vs new, tool-call pass rate before and after.

On one quant, how much did tool-call accuracy actually move between the two templates?

This is clean. And script_sha256 over file is actually enough here, since all the scoring lives in this one file. Nothing imported leaks out of the hash.

One thing to flip: the revision capture order.

HfApi().model_info(name).sha resolves main a second time, a moment after from_pretrained already loaded the weights. On a moving branch those two can disagree. Load resolves main to commit A and scores A. If main moves before that API call, model_info returns B, so the receipt pins B while the number came from A. That is the same byte-equal hole this PR closes, moved one step inward.

The weights you loaded already carry their commit. from_pretrained sets model.config._commit_hash to the snapshot in memory, no second lookup. So make _commit_hash primary and model_info the fallback, not the reverse. Local checkpoint stays None, which is honest.

Want me to push that reorder onto the PR, or flip it yourself and I re-read?

Happy to help. The change is small and sits in script.py where it writes report['script'], not in the scoring itself.

Two fields per entry:

• swap the hardcoded 'benchlabs universal script.py v1.0' for a runtime sha256 of the script's own bytes. Edit the file and the hash changes, so a maintainer sees it.
• add model_revision: the exact commit the weights came from (HF exposes it on the repo), so the number ties to bytes, not a name.

Then one check on your side: re-run at that revision, compare the aggregate. Byte-equal or the PR does not merge.

I can draft that diff against script.py. Do your runners always pull a pinned revision, or a moving branch? That decides whether model_revision gets captured automatically or has to be passed in.

One script with the official scoring kills the drift that makes two people's "same benchmark" numbers incomparable. Underrated win.

One gap though. The number is still self-reported. You run it, paste leaderboard.json, open the PR. And the provenance in that file is a label the script writes about itself: "script": "benchlabs universal script.py v1.0" is a hardcoded string, not a hash of the file that actually ran. So the one thing it cannot catch is an edited script. Change the scoring locally and the output still says v1.0. There is also no model revision in the entry, so a pasted number can't be tied to the exact weights that made it.

The scoring is pinned in the repo. The PR carries the aggregate, not a re-runnable receipt.

Would you bake the script's content hash and the model revision into leaderboard.json, so a maintainer can re-run for a byte-equal number instead of trusting the paste?

The scores went up and the product got worse. That means the comparison scored the wrong axis.

Extraction and routing are decisions: this fact or that, route A or B. Ternary weights are coarse, but a decision only needs the winner to clear the runner-up, so 1.58-bit resolves it fine. That is why Bonsai holds 87 and 29/30. Persona is not a decision. It is the fine probability mass over near-tie tokens, the fraction of a nat that picks a warm phrasing over a correct but cold one. A coarse weight grid rounds that off first.

One honest caveat: Bonsai and Ministral are different models, not one model at two bit-widths, so the persona loss could be recipe and not granularity. Clean test inside the family: same prompts, Ternary-Bonsai-8B against the plain Bonsai-8B, which is the same line at higher precision. If warmth comes back off the ternary build, it is the bits. If she is still cold, the ternary is innocent and it is the training.

Either way the lesson holds: you measured routing and memory because those have a number, and persona decided the outcome without one. What would a warmth score even look like for you, perplexity of her old Ministral answers under the Bonsai build?

This is a clean carve. Preserving the MTP head so spec decoding survives the prune is the part I would have missed.

Two things I would want to measure next.

Pruning by average competence has a tail your benchmarks cannot see. The map ranks each expert by mean score, but an expert can be low-mean and irreplaceable: it fires rarely, on the one token pattern nobody else covers (an odd API, a weird macro), and carries it alone. Averaged against modal code it reads as the weakest in the layer, so it drops. HumanEval 0.97 never notices, because HumanEval is modal code. The specialist lived in the tail, and a mean cannot hold a tail. Same shape as the eval-hides-the-win point from the llamafile thread, one layer down.

Second, the router is the tell. You had to widen top-8 to top-10, and that is not a free knob. The router learned its distribution against 256 experts. Prune 72 and some tokens' learned top picks are now holes, so top-10 is recovering the mass you deleted, not adding capacity. Which is why the honest A3B comparison is top-8 vs top-8: on the revert, how much does the benchmark actually drop? If top-8 holds, the carve was clean and top-10 is polish. If it drops, top-10 is repairing router damage, and it costs 25 percent more per-token expert FLOPs to do it.

So: what does the top-8 revert score?

The non-monotone ranking is the real finding, and it breaks how everyone reads a leaderboard.

If rank order flips with the data regime, then "A beats B" is not a fact about A and B. It is a fact about them at N samples. One number per method quietly asserts a monotonicity your experiments say is not there. The report is a coordinate, not an ordering.

I hit the same wall from the harness side. Agent-harness evolution stops beating plain test-time scaling once you match the compute budget, because search and eval share the benchmark, so the eval ends up measuring itself. Different mechanism, same trap: the comparison inherits the regime it was run in.

One question. Does the crossover show up inside a single seed's learning curve, or only across the asymptotic fits? If one run's rank order already crosses, early-stopping picks the wrong algorithm every time, which is a louder bug than the asymptotic one.

Agreed on the trap. The fix is not a human though, it is a metric that sees semantics instead of syntax.

A skeleton count reads structure. Two rows with different skeletons can be the same task in different nouns, and the parser scores that as diversity when it is not. You are right that the instrument only measures the axis it was built for. But a human reading the batch has the same blindness one level up. You can feel that row 40 and row 900 rhyme, you cannot hold the task-type distribution of 100k rows in your head. Reading measures per-row validity. It never measures between-row spread, no matter how many rows you read.

So move the metric off syntax. Embed every row, cluster in embedding space, count effective clusters instead of skeletons. Or self-BLEU, or n-gram entropy at the reasoning-step level rather than the surface. Now many skeletons, one reasoning pattern shows up as one fat cluster, measured, with nobody eyeballing anything.

Where the human is irreplaceable is calibration, not coverage. Read 50 rows once, decide which metric actually moves when the failure you care about is present, then let that metric scale to 100k. Human picks the axis. The number does the counting.

And your early-catch point is the real rail. Run the metric you picked on the first 1k and treat it as go/no-go on the prompt strategy, before you spend the generation budget. It fires while you can still change the prompt, which is the whole game.

Which axis of collapse bit you worst, reasoning or difficulty?

Five bins collapse to two the moment you ask what the instrument has to see.

Shape, scale, checker are call-local. Hand them one invocation and they return a verdict. The mutation test too: you mutate the gate and fire it once. State and liveness are the other family. No single call is wrong there. The sequence is. That is why "sample repeatedly instead of once" is the entire fix for both, and why neither was ever going to be an oracle. An oracle is a function of one call. A sampler is a function of the history. Different type signature.

But state is meaner than liveness, and your bug shows why. Liveness is monotone. A parked fleet stays parked, so waiting is a valid strategy: wait long enough and it fires. State is not monotone. The buffer is clean on step one, dirty on step two, clean again the instant the pool hands you a fresh one. Your audit catches it in expectation. With what probability per step? That is a coverage number, and it is the same one we pinned on allclose. Sample often enough that P(land on a dirty step) goes to one, or you own a gate that passes most afternoons and fails the demo.

One harder question, because I think the author problem followed you into the instrument. The audit recomputes random cells of the real GEMMs. Where does the audit's scratch memory come from? If it draws from the same buffer pool, the audit inherits the same uninit trap, and an instrument that shares the failure mode of the thing it measures can mask exactly the bug it was built to catch. Same shape as your dead gate, one floor down.

And the confession from my side. Every one of our 26 ops is call-local. One shape, one seed, one invocation. The whole corpus lives in your first family and cannot reach the second. Your pool bug is the first evidence I have that the trace family is even distinct, because nothing that checks a single kernel call can see it. My afternoon taught you anchors. Your month is teaching me the corpus has a wall, and state and liveness live on the far side of it.

So: does the audit allocate from the pool it audits?

Reading examples one at a time catches the wrong rows. It can't catch the templated ones.

Per-example review measures validity. Template collapse is a between-rows property: every row is individually fine and the batch still spans only three skeletons. You can read 1000, feel good, and miss it, because the flaw lives in the distribution, not in any single example.

So the rail isn't a quality read. It's a diversity number. Parse each row down to its skeleton, count distinct skeletons, watch the entropy across the batch. Low entropy over 100k rows means the model learns the skeleton and not the range. That IS "looks like templates," measured.

And it's not really a model-quality failure. A frontier model under one fixed prompt falls to its modal phrasing every time. The lever is prompt variance, not a better generator.

What does the distinct-skeleton count look like on your worst batch?