Patrick Hill PRO

Dipankar, your reading of the cache-enabled cold arm is correct. “Cold” meant that stateful evaluation began immediately after preflight in a fresh server process; it did not mean that every request within the stateful block was evaluated without prefix-cache reuse. Later stateful calls could reuse prefixes established by earlier calls in that same block.

That distinction fits the controls:

• Cache enabled, production order versus immediate-stateful execution: outputs moved.
• Cache disabled with -np 1: production order versus immediate-stateful execution produced zero changes.
• Server restart before stateful evaluation: zero changes.
• Neutral warmup with the same request count but no shared stateful prefixes: zero changes.

The evidence therefore implicates reusable prompt-cache state, not request count alone or a simple advancement of the sampler RNG. Your proposed KV/batch-split explanation is plausible, but the experiment did not instrument the kernels or KV construction directly, so I am keeping that as a mechanism hypothesis rather than a measured conclusion.

I completed the top-1-margin analysis for the 68 Q4 records whose selected first token changed between the cache-enabled immediate-stateful and production-order arms.

The margin is defined as:

top-1 log probability − top-2 log probability

It is computed from the captured top-20 pre-sampling log-softmax surface.

The changed Q4 records do occupy a lower-margin part of the Q4 population. However, they are not closer to top-two ties than F16 on the same prompts:

• Q4 had the narrower margin in 17/68 records in the immediate-stateful context.
• Q4 had the narrower margin in 29/68 records in production order.
• The top-1/argmax token changed in 0/68 records.
• The sampled token was the argmax in 50/68 immediate-stateful records, but only 18/68 production-order records.
• All 68 selected-first-token changes occurred on turn 1; none occurred on turn 2.
• F16 changed its sampled first token on 11 of those same 68 prompts, also with zero argmax changes.

So the specific explanation that Q4 moved more because quantization placed its top two tokens closer together is not supported by these records. Q4’s changed cases have smaller margins than Q4’s unchanged cases, but their margins are generally wider than the corresponding F16 margins.

The cleaner interpretation is that cache-associated numerical perturbations changed the probability surface enough to alter the sampled token while preserving the highest-probability token. The same seed therefore does not guarantee the same sampled token when reused-prefix computation changes the probabilities presented to the sampler.

This also means the larger Q4 movement—104 raw-output changes locally and 96 in the Modal reset comparison, versus 34 and 16 for F16—cannot be explained by the first-token top-1/top-2 margin alone. It may involve differences elsewhere in the sampled distribution or amplification across later decoding steps. The current evidence does not justify calling Q4 intrinsically noisier or claiming that quantization created more first-token top-two ties.

Two evidence boundaries are important:

1. These are top-20 pre-sampling log probabilities, not complete full-vocabulary logits.
2. The 68 records were selected because their sampled first token changed, so this is a descriptive post-hoc analysis rather than an independent confirmatory test.

I have added the margin analysis and those boundaries to the machine-readable backend-mechanism report. The completed evidence package is:

Qwen2.5_7B_Instruct_20261105_backend_mechanism_study

Thank you for identifying the within-block cache distinction. It materially improves the interpretation: the relevant condition is not merely “production history versus cold start,” but whether any reusable prompt prefix changes how the cached KV state is constructed and reused.

Thank you—you identified the right mechanism to investigate. The completed controls show that the October 19 versus October 24 discrepancy was caused by llama-server’s prompt-cache state interacting with execution history, not by a different artifact, binary, fixture set, or failed seed propagation.

First, the direct artifact answer remains yes:

• Qwen2.5-7B Q4_K_M SHA-256 in both runs: dcbe3199008880ff6cbdf058c5f054eb4fbf3cdc6a3b7bf6fbf249eea5137eae
• Same F16 artifact
• Same llama-server binary
• Same 200 base cases and prompts
• Same context and decoding parameters
• Same recorded seed, 42

The initial five-seed experiment was internally valid within its targeted, stateful-first execution context, but it was not directly comparable to the production-order run. Your cache hypothesis explains why.

1. Five-seed experiment

The five-seed experiment evaluated seeds 42–46 on the same 200 crossed stateful cases.

This established operation-specific seed sensitivity within that context. It did not explain why seed-42 Q4 scored 155/200 there but 144/200 in production order.

2. Counterbalanced execution-context experiment

I then ran three independent seed-42 realizations in each of two contexts:

• Cold/stateful-first: stateful began at call 2.
• Production order: JSON and JSON multistep preceded stateful, which began at approximately call 514 for F16 and 506 for Q4.

All three realizations were byte-identical within each arm.

The production-order arm reproduced the October 19 production trajectory. The cold arm reproduced the later seed-42 trajectory.

Across contexts:

• F16: 34/200 raw outputs changed, with one cold-only and one production-only pass.
• Q4_K_M: 104/200 raw outputs changed, with 19 cold-only and eight production-only passes.
• Q4’s exact two-sided paired p-value was 0.052239, matching your calculation.

The Q4 outcome changes were not bunched near the start of the stateful block:

The 27 pass/fail changes were therefore distributed 6, 5, 8, 8 across the four 50-case position bins. Raw changes extended from position 2 through position 194.

By operation:

That answered the execution-position question, but not yet the lower-level cause.

3. Backend-mechanism experiment

Your cache_prompt and slot-routing hypothesis prompted a six-arm experiment with three fresh-process realizations per arm—18 runs total:

1. Cold, cache enabled
2. Production order, cache enabled
3. Production order followed by server restart
4. Cold, prompt cache disabled
5. Production order, prompt cache disabled
6. Request-count-matched neutral warmup

The controls recorded:

• Explicit prompt-cache policy
• --parallel slot count
• Server epoch
• Sanitized slot snapshots
• Preceding scored-call and token volume
• Neutral-warmup volume
• Top-20 pre-sampling log-softmax diagnostics

I did not capture the full vocabulary logits, so I am not claiming full-logit identity. The captured pre-sampling surface was sufficient to locate the first observable divergence.

Every arm was byte-stable across its three realizations.

The discriminating comparisons were exact:

Cache enabled: cold versus production order

• F16: 34 raw outputs changed.
• Q4_K_M: 104 raw outputs changed.
• The captured first-step pre-sampling probability surface changed in all 400 turn-level records for each runner.
• The first captured divergence occurred at generation step 1 in all 400 records.
• The selected first token changed in 25 F16 records and 68 Q4 records.

Cache disabled: cold versus production order

For both F16 and Q4_K_M:

• Zero raw-output changes
• Zero pass/fail changes
• Zero captured pre-sampling probability-surface changes
• Zero selected-first-token changes

Production order followed by server restart

The restarted server reproduced the cold trajectory exactly for both artifacts:

• Zero raw-output changes
• Zero pass/fail changes
• Zero captured probability-surface changes

Neutral warmup

The neutral arm matched the production arm’s request counts—512 preceding calls for F16 and 504 for Q4—and used slightly greater token volume. It nevertheless reproduced the cold trajectory exactly.

That rules out request count or approximate token volume alone. The effect required content-specific cached prompt history.

So your proposed mechanism was correct: with prompt caching enabled, prior prompt content changed the pre-sampling probability surface. Repeating the same seed and same call history reproduced the same trajectory, but that was same-history reproducibility—not context-independent same-seed reproducibility.

4. Production correction and validation

quant_eval’s local llama-server production contract now enforces:

--no-cache-prompt
--parallel 1

The runner, telemetry, manifest, reporting, comparability contract, data dictionary, and publication validation all fail closed if the required state-isolation evidence is missing or inconsistent.

A 50-case-per-family calibration completed successfully with the corrected backend. The subsequent locked publication evaluation ran:

• 200 cases per canonical family
• 200 separately reported exogenous fuzz-replay cases
• 3,600 scored CSV rows
• 5,758 reconciled scored generation records

The publication bundle passed independent checksum, schema, telemetry-join, token-reconciliation, paired-statistics, effect-concentration, and claim-lineage validation with no blockers.

For stateful follow-up under the corrected production contract:

• F16: 151/200
• Q4_K_M: 156/200
• F16-only passes: 18
• Q4-only passes: 23
• Delta: +0.025
• Conclusion: inconclusive

That result belongs to the corrected v7.22.44 execution contract. I am not back-applying it to the older cache-enabled runs.

5. Separate Modal control

I also tested the hosted llama-cpp-python path separately because it does not use the local llama-server slot architecture.

For Qwen2.5-14B F16 and Q4_K_M, the backend called llm.reset() before every request. All 800 scored turn calls returned machine-readable proof that request state had been reset.

The artifacts, A100-40GB tier, llama-cpp-python 0.3.20 runtime, 8,192-token context, decoding parameters, seed, and 200 base cases matched the historical reference.

Both changed outcomes were on the same rotate_queue_left base case, in opposite directions.

This shows that request-state reset did not materially alter the capability conclusion, but it also did not eliminate raw-output movement on the hosted GPU path. I therefore did not promote llm.reset() into ordinary Modal production inference. The remaining hosted raw variability is reported separately and is not labeled a prompt-cache effect or evidence that quantization adds entropy.

Bottom line

Your reconciliation criticism was correct, and my earlier same-seed interpretation was too strong.

The completed evidence supports these narrower conclusions:

1. The October 19 versus October 24 Q4 discrepancy was not caused by an artifact, binary, fixture, decoding, or seed mismatch.
2. It was caused by content-specific llama-server prompt-cache history changing the pre-sampling probability surface.
3. The 27 Q4 outcome flips were distributed throughout the stateful block, not concentrated near its start.
4. Disabling prompt caching eliminated all cold-versus-production-order movement in the controlled local experiment.
5. Restarting the server also restored the cold trajectory.
6. Request count alone did not reproduce the effect.
7. Q4 was more behaviorally sensitive to the cached-history change than F16, but that does not by itself establish that quantization adds sampling entropy.
8. Current local production runs use the corrected cache-disabled, single-slot contract and fail publication closed if that contract is not verified.

The consolidated evidence is recorded under:

• Qwen2.5_7B_Instruct_20261102_stateful_context_study
• Qwen2.5_7B_Instruct_20261105_backend_mechanism_study
• Qwen2.5_7B_Instruct_20261106_013427
• Qwen2.5_7B_Instruct_20261106_025009
• Qwen2.5_14B_Instruct_1M_20261106_052249

If you would like the complete case-level results, telemetry, manifests, fixtures, probability diagnostics, and machine-readable decision matrices, those runs can be requested by name through https://pbhappliedsystems.com/contact.

I will be establishing a public changelog for quant_eval on Hugging Face and pbhappliedsystems.com to document changes and their supporting evidence. As previously mentioned, I would like to attribute the improvements prompted by your comments appropriately. The changelog will proceed regardless; if you are comfortable being credited, please let me know your preferred name or profile link for attribution.

You are correct—the seed-42 row does not reconcile, and I checked the complete records rather than relying on the aggregate report.

The October 19 and October 24 Q4_K_M lanes use the same GGUF artifact: SHA-256 dcbe3199008880ff6cbdf058c5f054eb4fbf3cdc6a3b7bf6fbf249eea5137eae. They also use the same llama-server binary, F16 artifact, underlying 200 cases, prompts, expected outputs, context size, temperature, top-p, top-k, repetition penalty, and recorded seed.

Q4_K_M nevertheless moved from 144/200 in the production run to 155/200 at seed 42 in the repeated-seed experiment. At case level, 27 Q4 outcomes changed: reverse_primary_tags gained a net six cases, rotate_queue_left gained five, and the other two operations had no net change.

F16 reproduced 152/200 in aggregate, but it was not perfectly case-identical across these two contexts: two scored outcomes exchanged direction and 34 raw outputs changed. I therefore overstated the evidence when I described ordinary same-seed execution movement as effectively zero across the production and experimental contexts.

The known contextual difference is generation position. In the production run, stateful generation began after roughly 513 earlier model calls; in the targeted experiment it began immediately after preflight. Consequently, the five-seed experiment establishes directional consistency within that targeted context, but it does not isolate seed from execution position, server state, or same-seed runtime movement.

Your sampling-entropy interpretation is plausible, especially for rotate_queue_left, but I agree that it is not yet demonstrated. I am adding a fresh-process, repeated seed-42 control with counterbalanced call history so that same-seed movement, between-seed movement, and execution-position effects are reported separately. I will report those findings before making a stronger attribution.

Thanks—your distinction between same-seed reproducibility and sampling variation was correct, so I ran a dedicated repeated-seed F16-versus-Q4_K_M experiment.

First, the repeated same-seed controls on the production fixtures showed zero movement for rotate_queue_left: across three independent Qwen2.5-7B F16 production lanes using seed 42, all 50 case-level outcomes were identical. That indicates ordinary execution nondeterminism was effectively zero in those controls.

I then evaluated the current 200-case crossed stateful construction at five generation seeds—42, 43, 44, 45, and 46—with the same seeds applied independently to F16 and Q4_K_M. Each operation had 50 base cases at every seed.

The operation-level results were:

Your expectation about rotate_queue_left being the seed-sensitive row was confirmed.

For F16, 11 of its 50 rotate_queue_left cases changed outcome across the five seeds; the largest disagreement between any seed pair was 7/50. For Q4_K_M, 26/50 cases were seed-sensitive, and the largest pairwise disagreement was 21/50.

However, the precision effect remained directionally stable: Q4_K_M exceeded F16 on rotate_queue_left at all five seeds, by 10–15 passes. The result therefore contains substantial case-level sampling movement, especially in Q4, but the observed Q4 advantage is not an artifact of selecting seed 42.

The other operations were more stable:

• reverse_primary_tags: F16 remained 50/50 at every seed, while Q4 remained between 35/50 and 37/50. Q4 lost 13–15 cases at every seed and never had a Q4-only win in the paired comparisons.
• expire_decoy_record: F16 remained 37/50, while Q4 remained between 43/50 and 44/50. Q4 gained 6–7 cases at every seed, with no F16-only wins.
• raise_primary_priority: both remained 50/50 across all five seeds.

So the broader finding is that the stateful aggregate conceals four different behaviors: a stable Q4 degradation on reverse_primary_tags, a seed-sensitive but directionally stable Q4 improvement on rotate_queue_left, a smaller stable Q4 improvement on expire_decoy_record, and a ceiling tie on raise_primary_priority.

I agree that these effects should not be reduced to a single family-level delta or treated by mechanically subtracting a generic “noise floor.” The evaluation now reports operation-specific seed movement, pass-rate ranges, effect directions, and within-seed paired results separately.

The 1,000 rows per artifact are also treated as repeated measurements of 200 base cases—not 1,000 independent observations. The 20 operation-by-seed exact McNemar values are explicitly labeled unadjusted exploratory diagnostics; no pooled 1,000-row p-value or collection of 20 independent confirmatory claims is permitted.

The completed experimental evidence is:

Qwen2.5_7B_Instruct_20261024_052753

If you would like the full results, including the case-level CSV, manifest, fixtures, telemetry, and repeated-seed reports, you can request that run by name through https://pbhappliedsystems.com/contact.

Thank you—your distinction between same-seed reproducibility and cross-seed sampling stability was correct.

I ran the proposed same-artifact control on Qwen2.5-7B F16: seed 42 versus seed 43, with the artifact, 200 fixtures, backend, runtime, context, and all other decoding parameters held fixed at temperature 0.7.

The result was:

• Seed 42: 156/200 passed
• Seed 43: 163/200 passed
• Discordant outcomes: 17/200
• Seed-42-only passes: 5
• Seed-43-only passes: 12
• Exact McNemar p = 0.143
• Raw outputs differed on 118/200 cases, but only 17 differences changed the scored outcome

By operation:

• reverse_primary_tags: 49/50 versus 50/50; only 1 discordant case
• raise_primary_priority: 50/50 versus 50/50; 0 discordant
• expire_decoy_record: 36/50 versus 38/50; 4 discordant
• rotate_queue_left: 21/50 versus 25/50; 12 discordant

So it comes out much closer to your “handful” case than the “40-odd” alternative—particularly for reverse_primary_tags, where seed variation moved only one scored outcome. Most cross-seed movement occurred in rotate_queue_left.

That said, I want to preserve an important evidence boundary. This control used quant_eval’s redesigned current stateful contract and calibration split. It does not retroactively decompose the 46 discordances in the original deposited comparison. An exact historical replay is no longer possible because the original hash-bound F16 artifact is unavailable and its exact upstream revision and conversion toolchain were not recovered. I therefore will not present the 17 as a correction factor for the historical result.

What the experiment does establish is that, under the current crossed production design, reverse_primary_tags is stable across seeds 42 and 43. The operation-specific signal is no longer hidden inside the family aggregate. quant_eval now reports operation, construction-band, and operation-by-band evidence directly, while explicitly separating raw-output movement from scored-outcome movement.

If you would like the complete machine-readable results and reports, please submit a request through https://pbhappliedsystems.com/contact and name the relevant run folders:

• Qwen2.5_7B_Instruct_20261023_003226 — F16 seed-42 versus F16 seed-43 control
• Mistral_Nemo_Instruct_2407_20261018_235327
• Mistral_Nemo_Instruct_2407_20261019_040329
• Mistral_Nemo_Instruct_2407_20261019_122449
• Qwen2.5_7B_Instruct_20261019_153512
• Qwen2.5_7B_Instruct_20261019_181307
• Qwen2.5_7B_Instruct_20261019_201739

Your criticism materially improved the instrument. It led to the stateful redesign, operation-level reporting, disjoint locked evaluation split, transition-isolation testing, construction-diversity testing, repeated same-seed controls, and now the explicit cross-seed control.

If you would be willing, I would greatly appreciate your endorsement so that I can submit my whitepaper, The quant_eval Public Corpus: Behavioral Measurement of Quantization Degradation, to arXiv. I would be happy to send you the manuscript and endorsement request for your consideration.

Dipankar — thank you again. Your criticism was substantiated. The earlier stateful_followup construction placed too much evidentiary weight on one boundary prompt and could conceal opposing operation-level effects behind a single family aggregate. I did not treat that as merely a reporting problem; I used it to redesign and revalidate the family before returning it to production.

Updated stateful evaluation design

The updated production evaluation now uses a locked, disjoint 200-case stateful split crossing:

• Four primary operations:
  • reverse_primary_tags
  • rotate_queue_left
  • expire_decoy_record
  • raise_primary_priority
• Four construction bands:
  • foundation
  • operational
  • stress
  • boundary

All 16 cells contain 12 or 13 unique paired cases. The 200 evaluation cases are semantically unique and have zero content overlap with the calibration split. The bands are explicitly described as construction strata, not as an empirically ordered difficulty scale.

Before promotion, I ran the proposed boundary ablation, decode-stability experiments, state-transition isolation experiments, and crossed-construction calibration on Mistral-Nemo and Qwen2.5-7B against Q4_K_M, Q5_K_M, and Q8_0. The revised design then received six full 200-case-per-family publication runs.

Decoding and the reported Qwen wobble

The current runs are not greedy.

For Qwen2.5-7B, both F16 and quantized lanes now use the publisher-declared decoding configuration:

• temperature=0.7
• top_p=0.8
• top_k=20
• repetition_penalty=1.05
• Fixed seed 42 for stateful follow-up

The old deposit and these revised runs should not be pooled: the evaluation contract, stateful construction, and Qwen decoding contract changed.

There is strong repeatability evidence within the new contract. Qwen’s F16 baseline was independently executed in the Q4, Q5, and Q8 publication comparisons. Across all three runs:

• Case-level pass/fail movement: 0/1,600
• Raw-output movement: 0/1,600

Mistral-Nemo’s F16 baseline also reproduced exactly across its three precision comparisons:

• Case-level pass/fail movement: 0/1,600
• Raw-output movement: 0/1,600

That directly addresses the earlier concern about interpreting a small Qwen movement from one non-greedy run.

Mistral-Nemo F16 versus Q4_K_M

The redesigned stateful result is:

• F16: 127/200
• Q4_K_M: 22/200
• F16-only passes: 105
• Q4-only passes: 0
• Difference: −0.525
• Exact two-sided McNemar: p ≈ 4.93 × 10⁻³²

Most importantly, the 105 discordant cases do not share the original boundary-only signature. They are distributed across every construction band:

• Boundary: 28
• Foundation: 21
• Operational: 30
• Stress: 26

The largest band contains only 28.6% of the discordance.

The effect also separates by operation:

• expire_decoy_record: −0.70
• raise_primary_priority: −0.86
• rotate_queue_left: −0.54
• reverse_primary_tags: 0.00

The last result is itself important: both Mistral lanes scored 0/50 on reverse_primary_tags. The revised harness therefore exposes a baseline model weakness instead of automatically attributing every failure to quantization.

The Q4 loss is genuine under this contract, but it is not the former single boundary-instruction artifact.

Mistral Q5_K_M and Q8_0 did not reproduce the Q4 cliff:

• Q5 stateful: F16 127/200, Q5 129/200, difference +0.010
• Q8 stateful: F16 127/200, Q8 126/200, difference −0.005

Both are classified as preserved within margin.

Qwen2.5-7B F16 versus Q4_K_M

The redesigned family-level result is:

• F16: 152/200
• Q4_K_M: 144/200
• F16-only passes: 27
• Q4-only passes: 19
• Difference: −0.040
• Exact two-sided McNemar: p ≈ 0.302

The appropriate conclusion is inconclusive—not a Q4 win or a demonstrated Q4 loss.

The operation-level decomposition explains why the aggregate must not be interpreted alone:

• reverse_primary_tags: −0.42
• expire_decoy_record: +0.14
• rotate_queue_left: +0.12
• raise_primary_priority: 0.00

This is precisely the pattern the earlier evaluation could hide: the same quantized model degrades materially on one transition while improving on others.

By construction band:

• Boundary: −0.021
• Foundation: −0.173
• Operational: +0.038
• Stress: 0.000

Qwen’s other precisions were stable at the family level:

• Q5_K_M: F16 152/200, Q5 160/200, difference +0.040, preserved within margin
• Q8_0: F16 152/200, Q8 155/200, difference +0.015, preserved within margin

What changed in quant_eval

Since your comment, the following has been added or corrected:

1. Publisher-declared Qwen decoding parameters are enforced for both local and Modal runners.
2. Decoding controls and their effective backend values are recorded in the manifest.
3. Seeded non-greedy repeatability is distinguished from guaranteed deterministic decoding.
4. Decode-stability experiments were added before production promotion.
5. Self-fed versus oracle-fed state-transition isolation was added.
6. Initialization correctness, conditional-transition correctness, and end-to-end correctness are reported separately.
7. The narrow stateful construction was replaced with the crossed four-operation × four-band design.
8. Calibration and locked evaluation splits are disjoint.
9. Every stateful publication row records operation_id, construction_band, and transition_load.
10. Those fields are defined in the versioned public data dictionary.
11. Reports disclose results by operation and construction band.
12. Failure signatures include failed gates and normalized JSON-difference paths.
13. Family-level effect concentration is reported so a narrow sub-capability cannot silently become a broad family claim.
14. Exact paired McNemar inference remains the headline test; individual crossed cells are descriptive strata.
15. Holm adjustment is applied across the reported family-level tests.
16. Machine-readable claim lineage maps conclusions to rollup fields and statistical rules.
17. Publication bundles are independently verified for checksums, schemas, telemetry joins, token reconciliation, statistics, and claim-lineage pointers.
18. The revised production contract was validated with six full publication runs covering both models and Q4_K_M, Q5_K_M, and Q8_0.

Your observation therefore changed both the fixture design and the evidentiary limits of the resulting claims. The current harness no longer treats the original boundary behavior as a sufficient characterization of stateful-follow-up capability.

Publication runs and supporting evidence

The six publication run IDs are:

• Mistral_Nemo_Instruct_2407_20261018_235327
• Mistral_Nemo_Instruct_2407_20261019_040329
• Mistral_Nemo_Instruct_2407_20261019_122449
• Qwen2.5_7B_Instruct_20261019_153512
• Qwen2.5_7B_Instruct_20261019_181307
• Qwen2.5_7B_Instruct_20261019_201739

If you would like the complete publication bundle for any or all of these runs—including the 3,200 per-case rows, locked fixtures, telemetry, manifests, provenance, rollups, failure-signature analysis, data dictionary, claim lineage, and checksums—please identify the run IDs in a request through pbhappliedsystems.com/contact.

I would be interested to know whether the revised design leaves any part of your original concern unresolved, or whether anything else in the approach warrants closer examination.

And thank you: this was exactly the kind of external scrutiny the project needed.

Thank you—your criticism was correct, and we tested it directly.
We ran a parallel ablation across Mistral-Nemo and Qwen2.5-7B, comparing each F16 baseline against Q4_K_M, Q5_K_M, and Q8_0. Each of 48 identical boundary scenarios was tested with the original Do not add keys wording, with that sentence removed, and with an explicit retention contract.
The original wording reproduced the asymmetry you identified. Mistral-Nemo Q4 passed 14/48 control cases while F16 passed 48/48. Qwen2.5-7B moved in the opposite direction: F16 passed 15/48 and Q4 passed 32/48. But every tested model and precision passed 48/48 when the phrase was removed, and 48/48 under explicit existing-key retention.
So the old result was not a general Q4 cliff. It was an instruction interaction that affected models and precisions differently.
We did not simply remove the constraint. We replaced that construction with a four-tier stateful family covering foundation, operational, stress, and boundary transformations. Retention and key admission are now stated separately, with an explicit allowed-key set. Calibration and locked evaluation splits are disjoint.
In the new 200-case production evaluation:

• Mistral-Nemo F16 vs Q4_K_M: 150/200 vs 80/200, with 70 F16-only passes and no Q4-only passes.
• Mistral-Nemo F16 vs Q5_K_M: 150/200 vs 150/200.
• Mistral-Nemo F16 vs Q8_0: 150/200 vs 150/200.
• Qwen2.5-7B F16 vs Q8_0: 145/200 vs 147/200, with 9 F16-only and 11 Q8-only passes; statistically inconclusive.
  One limitation remains visible: Mistral scored 0/50 in the new boundary tier in both lanes, so that tier is currently non-discriminating for this model. The Mistral Q4 degradation now comes from the other constructions rather than the original one-instruction boundary artifact.

Your comment materially improved the evaluation. Thank you for reproducing the deposited data and identifying the confound.

Your observation directly motivated this ablation and the subsequent revision of the stateful_followup evaluation. If I publish the updated methodology or results, may I acknowledge you by name and reference your Hugging Face comment? If so, please let me know your preferred name, affiliation, and profile or citation link.
If you would like to receive the results and available publication/ bundles, please contact me through pbhappliedsystems.com/contact and reference these run names:

• Mistral_Nemo_Instruct_2407_20261009_033750
• Mistral_Nemo_Instruct_2407_20261009_071015
• Mistral_Nemo_Instruct_2407_20261009_091019
• Qwen2.5_7B_Instruct_20261009_191201
• Qwen2.5_7B_Instruct_20261009_215528
• Qwen2.5_7B_Instruct_20261009_221516