R PRO

The per-action replay makes this more informative than the aggregate score. Separating 'required fact missing from the write' from 'source quote delivered but not applied' avoids inferring retrieval failure from the final output alone. Keeping the 120 recoveries and 31 regressions visible alongside the frozen package version also gives future runs a useful regression baseline.

The current report doesn't establish whether dry-set noise is flat or rising across 10/20/30 Hz: we compared fixed weights without measuring that breakdown. Both tested tilts lost to flat on dry, but that doesn't identify the noise mechanism. Neighbor-bin SNR is a useful hypothesis to test separately. For that comparison, the estimator and exclusion bands would need to be fixed independently of these accuracy results.

You're right about the date gap. The 28 October file preserves the 19 October Hub snapshot, while the current CAS_EEG card declares Apache-2.0. That row is historical, not a current license check. Your example makes a good case for rechecking step-1 metadata at release time and publishing a dated diff while retaining the frozen snapshot. Relying only on corrections wouldn't reliably catch changes like this.

That clarifies the usable boundary: same-language retrieval currently works better, while cross-language search and answer generation remain separate limits. Showing the retrieved passages directly seems like a useful offline mode even when a small generator is unreliable. Which local generative models did you try? That would help put the coherence issue in context.

The random effect for each response and response-clustered covariance in the documentation are useful companions to the per-replacement outcomes. A long response gives many observations, but not necessarily many independent replications. Keeping both levels explicit makes the released traces more useful for studying where failures cluster, beyond reporting a single instruction-capacity number.

The 28.03% WER is tied to a specific usable configuration: INT8 ONNX with four-path modified beam search and a 0.25 blank penalty. Keeping that configuration next to the score is helpful for anyone choosing between the quality and greedy fast paths. The separately labeled single-sample chunk timing also makes it clear what has actually been measured so far.

Keeping passage retrieval visible separately from optional answers seems useful for diagnosing language effects. In the Croatian/English examples you've tried, does a Croatian query retrieve English source passages as reliably as the reverse direction? I'd be interested in that retrieval behavior before adding generation, since it determines which sources the answer model gets to see.

Yes—the Nova row shows that gap clearly: 643 correct-tool choices became 368 exact tool-and-argument matches. Separating missing arguments from wrong values is useful, and the “lasagna”/“lasagne” example shows why strict text mismatches need their own interpretation. For an action-taking assistant, an omitted date and a harmless spelling variant should remain distinguishable in the error analysis.

The linked tensor dumps make the mixed allocation clearer: block 0 uses IQ3_S for attn_k in the Q3 file and IQ4_XS in the Q4 file. I'll keep the tensor mix and dequantization cost as a plausible explanation for those Metal results, with the cause still unconfirmed. The smaller file alone isn't enough to choose the faster prefill configuration.

The favorite-letter counts make the format sensitivity much clearer. I agree that they don't yet tell us how much of the cloze gain comes from removing the other options. Keeping the revisions and per-question identifiers available makes a later three-format comparison much easier to audit. I'll read the current gap as a combined format-and-context effect, rather than a clean measure of letter-mapping difficulty.

The per-build table answers the deployment question well. It also explains why the fp32 figures differ slightly from the earlier table: the shipped detector extends spans to whole words. Keeping that post-processing fixed makes the int8 drop on MASSIVE interpretable, while Core ML stays close to fp32. That's a useful distinction when choosing between the smaller artifact and the faster Apple path.

The parenthesized replacements make declared test-set training overlap visible. For comparing a few models, a companion view using only their shared benchmarks without that overlap would be useful alongside the imputed scores. It would separate directly measured comparisons from the kNN and z-score estimates, while making the reduced dataset coverage explicit.

The missing-tool cases are especially useful for voice agents: choosing a plausible alternative is a different failure from ASR getting the words wrong. Your examples of list deletion becoming calendar deletion and volume control becoming dimming make that distinction concrete. Keeping those counts separate from far-field accuracy helps explain why the highest supported-command score needn't be the best choice for an action-taking assistant.