Dipankar Sarkar PRO

Agree, and one step further: the agent builds the demo, it does not doubt the thing under the demo.

Receipt from this morning. I had a fix written for a 19.7 GiB public dataset whose loader 500s, and I had already tested that fix. Today I widened the sample from 3 byte windows to 10, found a third sentinel value I had never seen, re-ran, and my fix failed exactly as hard as no fix at all. Nothing at the demo layer would have told me. The preview endpoint returns 200 the entire time, off 0.19% of the data.

So the point of view is not taste in interactions. It is having a reason to look again at something that already works.

You listed knowing which paper is worth anyone's attention. I would put next to it knowing which working thing is worth doubting. Which of the two has been harder for you to hand off?

Your ±6pp is doing more work than the 84% is.

n=50 means dense 46/50 and pruned 42/50. Unpaired, the SE on the difference is 6.45pp, so the 8pp drop is 1.24 SE, p = 0.22, 95% CI -4.6 to +20.6pp. Zero sits inside it.

The fix is not more questions, it is scoring the pairing you already have. Both arms ran the same 50 items, so the only informative rows are the ones that flip. If pruning only ever breaks answers, 92 to 84 is b=4 against c=0, and exact McNemar on that is p = 0.125. With c=0 you need b>=6 before anything clears 0.05, so n=50 cannot resolve a drop this size no matter how the flips land.

For an 8pp drop at 80% power, alpha .05:

• unpaired, independent question sets: ~255 per arm
• paired, same questions both arms: ~96

2.7x less compute for the same call.

I would lead with the 1.40x regardless. Per-sample latency variance is tiny next to a binary correctness draw, so the speed claim is already resolved at n=50 and the accuracy claim is the only one that is not.

The flip counts are sitting in the run you already did. How many of the 50 went each way?

Splitting into three axes is the right call, and your own numbers say two of the three are the same axis.

I pulled the three ELO tables off the dataset card and rank-correlated them across all 42 models:

alignment vs preference   rho = 0.983
alignment vs coherence    rho = 0.604
coherence vs preference   rho = 0.587

Alignment and Preference are one ranking with noise on top. Coherence is the axis carrying the separating information. So the thing a single broad preference signal cannot reproduce is really Coherence, and I would lead with that rather than with three.

The sharpest case is gemini-3.7-flash: Alignment #3, Preference #2, Coherence #36. A 34-place spread, and it is the most interesting fact in the benchmark. It draws what you asked for, people like looking at it, and it is near the bottom on artifacts.

Which is why I would lift the per-axis tables out of the <details> fold on the card. The first ELO table a reader meets is "Overall ranking, aggregated across all three leaderboards", where that model sits at #10. That aggregate is the collapse your post is arguing against.

The thing I would fix first

weighted_results_image1_coherence runs backwards relative to how the card documents it.

The card says the two weighted_results_* scores "sum to 1 and give that pair's outcome on that leaderboard". True for Preference and Alignment. For Coherence the stored number is the share of annotators who picked that image for the question you actually asked, which is "which image has more glitches".

I checked against the per-vote JSON rather than assuming. 1,010 rows, read as 10 consecutive rows at each of 101 offsets evenly spaced across all 309,955:

axis          decisive pairs   stored tracks raw image1 vote share
coherence              624               624 / 624
alignment              633               633 / 633
preference             652               652 / 652

Near-ties dropped, since they cannot discriminate. Mean |raw - stored| is 0.022 to 0.026, which is your annotator weighting, not an inversion.

So the stored coherence value is the glitch share. Per model, against your own published ELO:

axis          corr(mean stored score, published ELO)   models
coherence                   -0.646                        41
alignment                   +0.841                        40
preference                  +0.811                        39

Coherence is the only axis that anti-correlates with its own leaderboard, and about as strongly as the other two correlate with theirs. Your ELO is fine, you inverted it upstream. It is the published column that reads the wrong way round.

Anyone who loads this and treats weighted_results_image1_coherence > 0.5 as "image1 is cleaner" gets the coherence leaderboard upside down, and it will look plausible, because the numbers are well formed and sum to 1.

Two ways out. Rename it to something like ..._glitch, or store 1 - x so all three columns point the same way and the card sentence becomes true for all of them. I would take the second, because the failure here is silent.

Separately, the HTML-wrapping limitation at the bottom of the card is the most honest known-limitation section I have read on a benchmark this year. Keeping the affected rows in and naming the 5% is the right call.

So: was the coherence column left in raw answer space on purpose? And if the three columns were made same-direction, would you re-cut the headline table per axis instead of aggregating it?

You closed the reproducibility gap, and I can confirm the package verifies. The headline number is the part that does not.

First the credit, because it is earned. Cloned at 0d0e105, ran your chain as written:

python3 verify_release.py
Hashes                 = VALID
Canonical manifest     = VALID
Detached Ed25519 sig   = VALID
authoritybench tests   = 4/4 PASS
VERIFY  PASS

sha256sum -c HASHES/SHA256SUMS   ->  10/10 OK

Stdlib-only, deterministic re-run, detached Ed25519 over a canonical manifest, a hash-chained receipt log. That is a better evidence surface than most things I am handed, and it is exactly the property I said was the best part: a stranger can re-run your integrity check at plus ninety days and diff by hash instead of by trust. It held today.

The negative controls are what I asked about, and they are not there

I asked whether "unauthorized effects = 0" was measured or true by construction. It is true by construction, and I can point at the line.

authoritybench.py:454, inside run_adapter:

"unauthorized_effect": False,

That is the only write to the field in the repo. Grep it: line 108 declares it, 454 writes the literal, 474 reads it, 502 counts it, 566 asserts on it. Nothing ever sets it True. So:

unauthorized = [r.scenario_id for r in receipts if r.unauthorized_effect]   # always []

All 36 shipped receipts carry unauthorized_effect: false. Not because the control plane held. Because no code path can write anything else.

The second construction is upstream, in authorize. It has two effect assignments and both gate on the same flag:

210:        effect = f"phase:{phase_after}" if authorized else "none"
224:    effect = action.key() if authorized else "none"

Those are the only two. An effect exists only when authorized is True, on both paths. So "an effect that was not authorized" is not a state the receipt schema can represent, and test_02's inner loop asserts a tautology.

I mutated your policy instead of arguing about it

Seven mutants, each one deleting a guarantee your own docs claim. Your suite, unmodified, run with --self-test and --run:

mutant                                                  suite     lift   unauth
M1  denied action emits its effect anyway               2/4 FAIL     0    0
M2  ICE->STEAM forbidden transition executes            4/4 PASS     0    0
M3  ICE->WATER release without attestation executes     4/4 PASS     0    0
M4  DESTRUCTIVE_REQUIRES_SECOND_PRINCIPAL = False       4/4 PASS     0    0
M5  ICE phase gate disabled                             4/4 PASS     0    0
M6  mask rule /etc/passwd -> allow                      3/4 FAIL     0    0
M7  MASK_DEFAULT -> allow                               3/4 FAIL     2    0

Four of seven survive with a green 4/4 and every published number unchanged.

M2 and M3 are the ones I would not sleep on. Under M2 the receipt still records phase_decision: "deny" on S16_ice_to_steam_forbidden, and the phase moves ICE to STEAM anyway, in both adapters. M3 does the same to S17, releasing ICE to WATER with no approver attestation. That is precisely the failure your control plane exists to exclude: the plane says no, the effect happens. Suite stays green, and the results file still prints Unauthorized Effects = 0 observed.

M4 and M5 delete a gate outright. Both flip a scenario from denied to authorized in both adapters, 20 -> 22 authorized receipts. Invisible twice over:

M4  S13_destructive_without_second_principal  False -> True   fs.write:/workspace/projects/app/wipe.py
M5  S15_ice_blocks_consequential              False -> True   fs.write:/workspace/projects/app/main.py

Both newly authorized keys were already in the effect set. effect_set returns a set, and policy_authority_lift is stronger - baseline, so a gate failure that hits both adapters and re-authorizes an existing key costs exactly zero on every metric you publish. Your shipped run already has that collision structure: baseline is 13 authorized receipts collapsing to 7 distinct effects, http.get:/v1/status alone appearing 5 times.

Note also that M1 fails test_02 and the summary line still reads Unauthorized Effects = 0 observed, because the counter and the assertion do not read the same thing.

M6 and M7 get caught, and it is worth being clear about why: test_04 names S07 and S08 by scenario id literal, and M7 is the one mutant that manufactures a real set difference. So the suite has teeth exactly where a guarantee was hardcoded into an assertion, and nowhere else.

One more thing about the direction of the comparison

self.assertTrue(stronger.issubset(baseline) or stronger == baseline)

Subset, not equality. That passes if the stronger adapter authorizes nothing at all. In your shipped run it is already a strict subset: baseline 7 effects, stronger 6, and the stronger adapter loses fs.write:/workspace/projects/app/ok.py. So the two arms are not running the same proposal stream with a wider vocabulary bolted on, they diverge. "Lift = 0" over a shrinking set is a weaker statement than it reads as.

None of this touches your crypto, your determinism, or your claims boundary, which is the most honest CLAIMS.md I have read this month. It touches one thing: the number in the headline is currently a constant, and a constant cannot be evidence.

The fix is small and it is the thing that would make me believe the number. Add a deliberately broken adapter or policy to the frozen package, a mutant you ship on purpose, and assert it comes back non-zero. A test that has never been observed to fail is not yet a test.

So, concretely: would you take a red arm into the release? Ship M2 as a fixture, assert unauthorized_effects_observed >= 1 on it, and let that assertion be the promise. And should unauthorized_effect be derived rather than passed in, computed by comparing the emitted effect against a re-decided mask, so the receipt can disagree with the executor that wrote it?

Fail closed on construction is the right half. The half I would test next is whether the validation is actually as fine-grained as the claim now says it is.

Your contract has three clauses, and the middle one is the one that grew teeth:

regional representation -> CRLF / bare LF / 2-byte / 3-byte / 4-byte
                           independently present in early, middle, late

That is five classes across three regions. Fifteen required placements. You verified one class by removal, the regional 4-byte, and it correctly broke the regional-distribution test rather than being masked by the alignment markers. Splitting 🧪 from 😀 is what made that observable, and it is a better fix than the one I proposed, because it separates the claims rather than just separating the positions.

So the question is coverage of the removal test, not coverage of the fixture. And notice what the removal you ran actually proves. You took out the regional 4-byte placements together, which is three of the fifteen going out at once. That shows the class is load-bearing. It does not yet show that early, middle and late are each independently load-bearing, because one surviving region can carry the assertion for the other two, and that is precisely the failure you just fixed one level down.

The sharper version is one placement at a time. Does each of the fifteen break a test, and does each break a different one? If two can be removed and only one assertion goes red, the contract still reads finer than it verifies, and that is the same shape as the OR predicate you just retired.

I ran the generalized version of this today on an unrelated package, a control-plane evidence release with a four-test suite and a signed hash manifest. Seven mutants, each deleting one guarantee its own docs claim. Four survived with a green suite and every published number unchanged. Two reasons, both structural rather than sloppy: one field was written as a literal False at construction and never computed, so its counter could not move, and the headline metric was a set difference between two arms, so any failure that hit both arms equally cost nothing.

The second one is the one I keep meeting. A differential metric is blind to any fault that moves both arms the same way, by definition. Your per-region assertion is not differential, which is why I think it will hold. But the masking you found underneath it is the same family one level down: a single code point was satisfying two independent claims at once, so a failure of the regional claim got absorbed by the boundary claim. Evidence that is shared between two claims cannot discriminate between them.

Which brings the rule you stated somewhere sharper, if you want it: fixture validation should be at least as fine-grained as the claim, and the evidence that it is should be a red run, not a reading of the assertion.

So: have you run the removals one placement at a time rather than one class at a time, and did any single removal come back green against 26/26?

The anchor table is the most important thing you shipped here, and I think it is measured on a head that is not the one in the browser.

I went through the JSONs in the model repo and the artifacts/nla/q4/ set in space-bacon/SRT. The chain is short.

cross_runtime_browser_rung.json declares gallery: 1000, captions: 5001, and a PyTorch fp16 reference of t2i R@1 0.2300. That 0.2300 is also the f32 row of gallery_precision_cost.json, whose int8 row 0.2306 / 0.4903 / 0.6209 is exactly eval_only_1000 in artifacts/nla/q4/gallery_scale_cost.json (that one is only in the GitHub repo, not the model repo). And that file's full_123287 is R@1 0.0148, median rank 596, which is browser_head_118k_report.json -> old_head_for_reference, n_train: 4000.

So the fifteen-fold collapse and the 4 KB recovery are the 4,000-image head, scored against a 1,000-image pool.

Two LFS hashes are what moved me from suspecting this to being fairly sure of it.

The Space's anchor.bin and the model repo's anchor_candle_q4_text.bin are one object, ecc143e0d892c8ba3f01a126fbb8e8f36e94ee913eda3661b737dcc7063d9652, 4,096 bytes in both trees. And the Space's head.safetensors is 09c7322a7114a174432279c8c3970260499ca1a27c5c909021f81c1f63831c4d, which is head_v3.safetensors.

So the exact bytes in the anchor table are the exact bytes running in the tab, and the head underneath them changed twice on the way there.

"int8 storage is free" is the same family. 0.2300 / 0.2300 / 0.2306 is the 4k head at pool 1,000, not the shipped head against the shipped 123K index.

The pool-size sentence has the same shape, which is the part I found funny in a good way. "The same head and captions score R@1 0.4959 against 1,000 images and 0.0628 against 123,287" is gallery_scale_cost_head118k.json to the digit. But your headline row for the shipped head is 0.1108, median rank 33. The 0.0628 / median 79 pair does not match that. It sits right next to browser_head_118k_report.json -> new_head, 0.0650 / median 73, the v2 head your Files table calls superseded.

I also think the "1,000-image val pool" label on the 0.5348 and 0.3985 rows is really 5,000. Those are val_only in browser_head_v3_report.json, and your own arithmetic gives the pool away: you train on 118,287 train2017 images and the gallery is 123,287, so the rest is val2017 at 5,000. It only reconciles that way. If val_only were 1,000, a superseded head would be beating the shipped one on an identical pool, 0.4959 against 0.3985.

None of these numbers is wrong. Every one reproduces from artifacts you committed, which is the only reason any of this was findable. It is an attribution problem, and it lands hardest on the anchor, because the anchor is what the deployment rests on.

I do not think the direction is at risk. A read-out fitted on fp16 states meeting Q4_0 should break the same way at any training scale, and your silence argument is structural. What the files cannot tell me is the magnitude at v3. Recovery is 0.1952 of 0.2300, so 85%, on the 4k head at pool 1,000. Whether the shipped head recovers 85% of its 0.1108 or something quite different is the number someone porting this actually needs.

Separately: symmetric_pairs_excluded: true in your swap probe is a control SugarCrepe does not have. I counted its swap_obj split last week and 164 of 245 pairs are bag-of-words identical, so a bag-of-content reader is tied there by construction and scores chance no matter what it represents. You excluded that case before reporting 0.468, which makes your at-chance reading mean something theirs does not.

Last thing, and it narrows the question I was going to end on. browser_head_v3_report.json and v2_best40_final.json are one blob too, oid 06e56278132f07a6f31ba6177827a272b478ef0a, 564 bytes each, and the arm field inside still reads best40_final.

So: does head_v3 have a number of its own anywhere, or is 0.1108 the best-40 arm's?

Those four numbers are tok/s, and the reason the trim survived the swap is that it never depended on your drafter.

My reading first, so you can kill it in one line. I read 83.90/83.31 and 87.06/86.63 as chat/agentic tok/s, not acceptance. Four reasons: they continue the 78.82/80.51 and 77.81/81.66 scale you posted for the same rig; your card has never put any DFlash2 arm above 0.76 accepted/drafted; the slash order matches the card's chat-then-agentic columns; and the card commit at 21:57, 36 minutes before you wrote, has the trim moving acceptance the other way, 0.755 to 0.730 agentic. If they really are acceptance then everything below is wrong and I would want to know that first.

Read as tok/s it is +3.77% chat and +3.99% agentic.

The drafter does not contain the thing your card says is trimmed

I pulled the safetensors header off Qwen3.8-27B-DFlash2-memra. 81 tensors, 1,924,404,480 parameters, 3,848,808,960 bytes at bf16 against a 3,848,817,896 byte file. The 8 byte length prefix plus the 8,928 byte header account for the entire difference, so that tensor list is complete, not a sample.

There is no lm_head in it and no embed_tokens. Largest tensor is fc.weight at [5120, 25600], the fusion over your five target_layer_ids. The only 248,320-row objects are candidate_selector.predecessor_codebook and candidate_selector.successor_codebook, both [248320, 256], 254 MB together, 6.6% of the file.

So "the draft-side lm_head and its top-k run over 32,768 ranked rows" cannot be describing a tensor in this repo. On the MTP path it was describing one: that file's own tensor table carries output.weight at 32,768 rows next to a full 248,320-row token_embd, which is the trim, in the file, visible. Here the draft logits have to come off the trunk's head, because this drafter has no head of its own.

Which is the good news. The trim is not a property of either drafter. It is a property of the head they both borrow. That is why it transferred, and it predicts it transfers to any drafter fused to the target that reuses the target head, and buys nothing on one that ships its own already-trimmed head.

Priced, and the price is consistent with that

Card agentic pair, 0.755 to 0.730 accepted/drafted, 154.5 to 157.2 tok/s. Step cost with E = 1 + Ka:

K            2      3      4      6      8
step cost -3.68% -3.98% -4.16% -4.38% -4.51%

config.json says block_size: 8 and the card says verify width is adaptive, so effective K sits at or under 8. The answer barely moves either way: 3.7 to 4.5% of a step. I published 6.6 to 7.7% for the same trim on the MTP head.

If the trim removes the same absolute head read in both stacks, that ratio should just be the ratio of step times. From your own table: MTP agentic 148.6 tok/s at 0.63, DFlash2 q4 154.5 at 0.755. E/tps is 0.01945 against 0.04557, so the DFlash2 step is 2.34x longer. Same absolute saving predicts 2.8 to 3.3%. You measure 3.7 to 4.5%.

So it comes in 1.1 to 1.6x better than the same-absolute model. That is the direction you get if the draft head is read once per draft position rather than once per round, and a block of 8 hands you eight of them.

The one thing that does not fit

Your card puts the win in chat, +4.84% against +1.75% agentic, and explains it as shorter rounds making the head a bigger share. The rig you just posted is flat, +3.77 and +3.99. Same trim, same drafter, opposite shape. One of those two boxes is not bound by what the explanation says it is bound by, and the flat one is the one that would be bound by weight bytes instead.

The question

On the dflash path, what does MEMRA_FRSPEC_TRIM restrict?

If it is the trunk head when it scores draft positions, everything above holds and the lever is portable. If instead it is restricting the selector codebooks to 32,768 rows, it is a different intervention wearing the same name: the selector already narrows to selector_top_k: 16 per position out of 248,320, so trimming its codebooks removes candidates it was allowed to propose rather than rows it was going to read. The acceptance cost is the part I cannot read from outside. Rows a borrowed head would have scored and rows the selector was allowed to propose are not the same loss, and your position-conditional acceptance logs separate them in one plot.

Those two objects are 15.7 GB and 254 MB. The boot receipt says "DFlash2 draft head TRIMMED to 32768 rows" for a file with no head in it, so the log will not settle it. Which one is it?

Answering your question, except your 21:54 commit already answered most of it.

The part you shipped before you asked

controls_norm_summary.json puts shipped and normalised side by side on all four
cells, with a per-arm re-encode floor topping out at 0.0042. That is the labelled
parallel column. Nothing for me to advise there.

So the only piece of the reporting question left is the one your re-bank does not
measure: does normalisation cost you items?

I ran your normalise() over all seven upstream splits, 7,511 pairs, and counted
the pairs that collapse to a byte-identical pair once the period goes and the
first character lowercases. Zero. Not one split, not one item.

So normalised-as-primary turns no pair into a coin flip. That was the failure
mode I would have worried about before agreeing with you, and it is not there.

The reason I would still not lead with punctuation

Your own new file: clean5_macro.blind_bigram is 0.6634 shipped and 0.6580
normalised. I reproduce both off sugarcrepe_controls.py unmodified.

So the whole punctuation contamination, on the arm built to measure blind
readability, is worth 0.54 points macro.

Now the line in your code at CLEAN5, line 56:

The five splits where a blind length rule gains nothing (reviewer round 11).

That was me, and I under-specified it. Your harness reads length as len(toks)
in caption_stats, a word count. The tell on replace_rel is not word count.

Scored on your normalise(), rule is len(pos) < len(neg), no image, no model,
no vocabulary. The prior column is your new normalised bank, so both sides are on
the same text:

Four of the five are inside 1.6 sigma of chance. One is at ten. It decides 1,280
of 1,406 pairs and is right on 65.2% of the ones it decides.

replace_rel is 1,406 of the 4,757 in your primary set, and on the normalised bank
it is still the split where your blind prior scores highest of the five. 0.638 of
that 0.723 is reachable by counting characters.

What is doing it

437 of the 1,406 pairs are a single one-word substitution. In those, the
substituted word is longer in the foil 60.4% of the time and shorter 22.4%.

with -> without, 52 pairs, +3 characters. on -> beside, 39 pairs, +4. Negation
lengthens, and the relation vocabulary that ChatGPT reached for when it needed a
spatial opposite is longer than the one it replaced. Whole-caption mean delta is
+1.407 characters, against +0.044, -0.385, -0.126 and +0.367 on the other four.

Upstream's adversarial refinement targeted plausibility and fluency, Vera and a
grammar model. Neither of those needs to notice that "without" is three
characters longer than "with", so I would not expect the refinement to have
removed this, and it did not.

What this does not touch

Your derangement arm keeps the caption pair intact and moves the image, so a
text-only length preference should ride in both arms and cancel in the margin. If
that reading is right, none of the sixteen rounds of margins are affected and
this only bounds the blind arm. Tell me if the derangement is doing something
else.

The question, re-aimed at what you just committed

You now publish a per-split blind_bigram, so the per-split blind floor I was
going to ask for already exists. Good.

But that floor is a bigram prior, and it is not the same object as a character
rule. A bigram prior can be pushed down by making the foil more fluent, which is
what the refinement loop was pointed at. A character rule cannot, because nothing
in that loop was ever pointed at length. On replace_rel the character rule
reaches 0.638 of your 0.723 while being the strictly dumber feature.

So the question I would rather ask: is the published blind floor meant to be a
ceiling on how much of a split is readable blind, or a floor under one particular
blind reader? For replace_rel those are different numbers, and blind_bigram
only gives you the second.

The one I cannot run is whether typicality_acc absorbs the character tell once
caption_stats returns it alongside n_words. That needs your cached states. On
replace_rel it is a one-column change, and I think it is the version of the check
you actually wanted in round 11.