Post
824
๐ The attention mask stopped being an audit.
An autoregressive model must not let position t depend on anything after t. Everyone checks this by inspecting the causal mask โ but hybrid stacks now mix attention with state-space scans, and a scan has no mask. Every mask can be correct while information leaks through scans, aggregations, or normalization.
โ๏ธ So we test the property directly. Two inputs identical except at the last position, two forward passes, compare each layer's prefix, report the first layer that moves. No training, no gradients, no accelerator โ seconds on CPU.
๐ Across 192 injected faults on eight checkpoints, mask inspection detected 0. The per-layer audit localized 192/192 to the exact layer.
๐ฏ Then we read the source before running anything. In transformers 5.7.0, the reference chunked scan reduces the inter-chunk recurrence over the input chunk axis; zamba2 and nemotron_h reduce over the output chunk axis. One axis. The dynamic audit confirmed the prediction exactly: Zamba2-1.2B leaks from length 256, its declared chunk size, and Nemotron-H-8B from 128, its declared chunk size. Bamba, Falcon-H1, Granite-4.0-H, Mamba2 and RecurrentGemma came back clean.
โ ๏ธ Scope: the defect is on the PyTorch chunked-scan path, which runs whenever the fused kernels are absent โ CPU, CI, stock installs. We could not build those kernels, so the fast path is untested and open. That caveat cuts both ways: a model can pass every fused-kernel test and still leak the moment it runs without them.
๐งช AX-RAY now carries this as its own axis. 39 models scored across causal, white-box and behavioral axes: 21 A, 3 B, 1 C, 14 F โ with exactly 2 Causal-LEAK verdicts, the two the paper predicted. Badges separate a weights-level audit from an API-only one, so the two never get read as the same claim.
๐ https://arxiv.org/abs/2608.22876
๐ฌ FINAL-Bench/AX-RAY
๐ค The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models (2608.22876)
An autoregressive model must not let position t depend on anything after t. Everyone checks this by inspecting the causal mask โ but hybrid stacks now mix attention with state-space scans, and a scan has no mask. Every mask can be correct while information leaks through scans, aggregations, or normalization.
โ๏ธ So we test the property directly. Two inputs identical except at the last position, two forward passes, compare each layer's prefix, report the first layer that moves. No training, no gradients, no accelerator โ seconds on CPU.
๐ Across 192 injected faults on eight checkpoints, mask inspection detected 0. The per-layer audit localized 192/192 to the exact layer.
๐ฏ Then we read the source before running anything. In transformers 5.7.0, the reference chunked scan reduces the inter-chunk recurrence over the input chunk axis; zamba2 and nemotron_h reduce over the output chunk axis. One axis. The dynamic audit confirmed the prediction exactly: Zamba2-1.2B leaks from length 256, its declared chunk size, and Nemotron-H-8B from 128, its declared chunk size. Bamba, Falcon-H1, Granite-4.0-H, Mamba2 and RecurrentGemma came back clean.
โ ๏ธ Scope: the defect is on the PyTorch chunked-scan path, which runs whenever the fused kernels are absent โ CPU, CI, stock installs. We could not build those kernels, so the fast path is untested and open. That caveat cuts both ways: a model can pass every fused-kernel test and still leak the moment it runs without them.
๐งช AX-RAY now carries this as its own axis. 39 models scored across causal, white-box and behavioral axes: 21 A, 3 B, 1 C, 14 F โ with exactly 2 Causal-LEAK verdicts, the two the paper predicted. Badges separate a weights-level audit from an API-only one, so the two never get read as the same claim.
๐ https://arxiv.org/abs/2608.22876
๐ฌ FINAL-Bench/AX-RAY
๐ค The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models (2608.22876)