MiMo-V2.6-Flash — REAP-50 (MXFP4 + FP8 Master Checkpoint)

85.8 GiB bit-exact 50% REAP expert-pruned master checkpoint of XiaomiMiMo/MiMo-V2.6-Flash (315B → 164B total, 11B active) in native OCP MXFP4 + FP8 + BF16.

Sibling Artifact	Format	Size	Description
`MiMo-V2.6-Flash-REAP50` (this repo)	Native MXFP4 + FP8 + BF16 (`safetensors`)	85.8 GiB	Bit-exact 128-expert master checkpoint (with vision/video/audio encoders + 3 MTP layers)
`MiMo-V2.6-Flash-REAP50-GGUF`	GGUF (`MXFP4_MOE`, `Q3_K_S`, `Q2_K`, `mmproj`, `mtp`)	64.8–86.7 GiB	Single-file GGUFs + multimodal projector + 3-layer MTP draft head

MiMo-V2.6-Flash-REAP50 is a 50% expert-pruned version of XiaomiMiMo/MiMo-V2.6-Flash (315B total / 11B active omni-modal MoE) produced using REAP (Router-weighted Expert Activation Pruning, arXiv:2510.13999), reducing each of the 59 MoE layers from 256 routed experts to 128 routed experts while keeping top-8 active routing unchanged.

Total checkpoint size drops from 157 GiB → 85.8 GiB (164B total parameters, 11B active parameters), allowing the full model (including its vision/video/audio encoders and 3-layer MTP speculative decoding head) to fit comfortably in a single 128 GB unified-memory system (NVIDIA Jetson AGX Thor, DGX Spark, Mac Studio, or AMD Strix Halo).

✨ Highlights

Bit-Exact Native OCP MXFP4 Survivor Weights (85.8 GiB, Down from 157 GiB): All 128 surviving routed experts per MoE layer retain their exact upstream packed OCP MXFP4 (weight_packed U8 + weight_scale F8_E8M0) bytes without re-quantization.
Unmodified Multimodal Encoders, Attention & 3-Layer MTP Head: Keeps Layer 0 (dense MLP), all 60 hybrid MLA attention layers (full + sliding-window SWA in FP8 E4M3 + F32 block scales), the vision (vision_tower), video (video_tower), and audio (audio_tower) encoders + multi-modal aligners (BF16), and the 3-layer Multi-Token Prediction (mtp) head 100% bit-identical to the base checkpoint.
6-Bucket Domain-Balanced REAP-50 Pruning: Calibrated across 298 sequences (610,304 tokens) spanning English general/encyclopedic, software engineering & code, competition mathematics, Chinese (zh), agentic tool-use, and synthetic omni-modal embeddings, with union top-16 specialist protection per domain (35–75 specialists locked per layer).
Single-Box 128 GB Fit: Leaves ~42 GiB of unified-memory headroom on a 128 GB Jetson AGX Thor or DGX Spark for KV cache and multimodal activations.

📋 Model Summary

Property	Base (`XiaomiMiMo/MiMo-V2.6-Flash`)	This Model (`MiMo-V2.6-Flash-REAP50`)
Total Parameters	~315B	~164B
Active Parameters / Token	~11B	~11B (unchanged)
Decoder Layers	60 (Layer 0 dense + Layers 1–59 MoE)	60 (Layer 0 dense + Layers 1–59 MoE; `./config.json`)
Routed Experts per MoE Layer	256 (`n_group=8`, `topk_group=4`)	128 (`n_group=8`, `topk_group=4`, 16 experts/group)
Active Experts per Token (`num_experts_per_tok`)	8 (`routed_scaling_factor=2.827`)	8 (`routed_scaling_factor=2.827`)
Expert Precision	Native MXFP4 (`E2M1` + `E8M0` scale, block=32)	Native MXFP4 (bit-exact slice, zero re-quantization)
Attention Precision	Native FP8 (`E4M3` + `F32` block scale 128×128)	Native FP8 (bit-exact copy)
Vision / Video / Audio Encoders	BF16 (`vision_tower`, `video_tower`, `audio_tower`)	BF16 (bit-exact copy, unpruned; `./audio_tokenizer/model.safetensors`)
MTP Draft Head (`mtp.layers.0..2`)	3 dense+attention layers (FP8/BF16)	Preserved bit-exact (`./model_mtp.safetensors`)
Total Checkpoint Size	157 GiB	85.8 GiB (64 EP shards `model_pp0_ep0..63_shard0.safetensors` + `./model_mtp.safetensors`; `./model.safetensors.index.json`)
License	Apache-2.0	Apache-2.0

Evaluation Status Note (2026-10-24)

Surviving weights are 100% bit-exact slices of XiaomiMiMo/MiMo-V2.6-Flash. No post-pruning router ridge-regression or weight modification was applied (e_score_correction_bias and gate.weight for the 128 kept experts are copied verbatim from the base checkpoint; see scripts/apply_mask.py and wiki/98-router-kd-global.md).
Teacher-vs-student KL / NLL benchmark status: A paired held-out teacher-vs-student KL/NLL table (of the kind published on GLM-5.3-Flash-REAP50-FP8-v2 and DeepSeek-V4.1-Flash-1017-REAP192-MXFP4) has not yet been run for MiMo-V2.6-Flash-REAP50 because MiMo-V2.6-Flash's custom mimo_v2_flash architecture (./modeling_mimo_v2.py) is not yet wired into our standalone teacher-logit evaluation harness. See the companion MiMo-V2.6-Flash-REAP50-GGUF card for additional details on GGUF conversion and runtime support.

What Changed vs. the Base Model

Preserved Bit-for-Bit (Zero Re-quantization)

Layer 0 (Dense MLP + Full MLA Attention): 100% untouched.
All 60 Attention Layers (self_attn + layer_scale): Full MLA layers (0, 5, 11, 17, 23, 29, 35, 41, 47, 53, 59) and Sliding-Window MLA layers (all others, sliding_window_size=512) are copied byte-for-byte in native FP8 (F8_E4M3 + F32 weight_scale_inv).
Multimodal Towers (vision_tower, video_tower, audio_tower, multi_modal_projector): 100% untouched in BF16 (plus ./audio_tokenizer/model.safetensors).
MTP Speculative Decoding Head (mtp.layers.0..2): 100% untouched in ./model_mtp.safetensors.
Surviving 128 Routed Experts per Layer: Their MXFP4 weight_packed (U8) and weight_scale (F8_E8M0) bytes are sliced directly from the base tensors without dequantizing or re-quantizing (scripts/mimo_shards.py).

Pruning Method

1. Group-Balanced REAP Saliency (`n_group = 8`, `16` experts kept per group)

MiMo-V2.6-Flash uses grouped sigmoid top-k routing (n_routed_experts = 256, n_group = 8 groups of 32 experts, topk_group = 4, num_experts_per_tok = 8, routed_scaling_factor = 2.827; see research/MIMO_V26_FLASH_SURVEY_2026-09-23.md). Because the router reshapes experts into [n_group, n_routed_experts // n_group], pruning exactly 16 of 32 experts within each of the 8 routing groups preserves the exact group topology (n_group = 8, 16 experts per group) so existing inference kernels (vLLM, SGLang, Transformers, llama.cpp) work out-of-the-box with only n_routed_experts: 128 updated in ./config.json.

For every MoE layer $l \in {1, \dots, 59}$ and expert $j \in {0, \dots, 255}$, we streamed 610,304 calibration tokens layer-by-layer on an NVIDIA Jetson AGX Thor (128 GB unified memory) via scripts/mimo_saliency.py and scripts/calib_pass.py, executing the exact MiMoV2FlashMoE group-topk routing and dequantizing the selected MXFP4 experts on the fly to compute the REAP saliency score (arXiv:2510.13999):

$S_j = \frac{1}{|T_j|} \sum_{t \in T_j} g_{t,j} \cdot \left\| W_{\text{down},j}\!\left(\text{SiLU}(W_{\text{gate},j} h_t) \odot (W_{\text{up},j} h_t)\right) \right\|_2$

2. Six-Domain Calibration & Specialist Protection

Calibration data (298 sequences × 2048 tokens = 610,304 tokens; defined in scripts/mimo_corpus_spec.py and wiki/91-capability-coverage.md) was balanced across 6 buckets:

general — English encyclopedic & long-form analytical text
code — Multi-language software engineering, algorithms, systems code
math — Olympiad & undergraduate mathematical proofs and derivations
zh — Simplified & Traditional Chinese technical and literary text
tool — Multi-turn agentic tool-calling, JSON schemas, and reasoning traces
multimodal — Text sequences interleaved with synthetic vision/audio projector embeddings matching the empirical norm/distribution of multi_modal_projector outputs (scripts/media_loaders.py)

Within each layer:

The top-16 experts per domain (by per-domain REAP score $S_j^{(d)}$ with $f_j^{(d)} \ge 16$) are locked as protected domain specialists (35 to 75 specialists locked per layer; scripts/reap_select.py).
Remaining slots in each of the 8 routing groups (32 → 16 experts per group) are filled by cross-domain normalized REAP score (with a frequency floor $f_j \ge 40$).

Companion GGUF Repository

For llama.cpp inference (MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf, MiMo-V2.6-Flash-REAP50-Q2_K.gguf, plus multimodal mmproj and mtp GGUFs), see patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF.

🔎 Provenance & Verification Ledger

Every claim, architectural parameter, and pruning step on this card can be audited directly against the files in this repository and the companion open-source pipeline:

Claim / Artifact	Primary Verification Source
Upstream Base Model (`315B` MXFP4+FP8, Apache-2.0)	`XiaomiMiMo/MiMo-V2.6-Flash` · `./README_upstream.md`
REAP Pruning Formulation	Lasby et al., 2025 — REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression (`arXiv:2510.13999`)
Model Config, Modeling Code & Shard Index	`./config.json` · `./modeling_mimo_v2.py` · `./configuration_mimo_v2.py` · `./model.safetensors.index.json`
MTP Draft Head & Audio Tokenizer Weights	`./model_mtp.safetensors` · `./audio_tokenizer/model.safetensors` · `./audio_tokenizer/config.json`
MiMo-V2.6-Flash REAP-50 Pipeline Repository	`patrickbdevaney/xiaomi-2.6-flash-REAP` (`main` @ `329123f`) · `wiki/README.md` · `wiki/00-log.md`
Architecture Survey & SOTA REAP Methodology	`research/MIMO_V26_FLASH_SURVEY_2026-09-23.md` · `research/REAP_METHODOLOGY_SOTA_2026-09-23.md`
Saliency Streaming, Selection & Bit-Exact Shard Slicing	`scripts/mimo_saliency.py` · `scripts/reap_select.py` · `scripts/apply_mask.py` · `scripts/mimo_shards.py`

Citation & Acknowledgments

Base Model: Xiaomi MiMo Team — XiaomiMiMo/MiMo-V2.6-Flash (Apache-2.0 License)
Pruning Algorithm: REAP: Router-weighted Expert Activation Pruning (arXiv:2510.13999)
REAP-50 Pipeline: patrickbdevaney/xiaomi-2.6-flash-REAP

Downloads last month: 513

Safetensors

Model size

159B params

Tensor type

BF16

F8_E4M3

F32

Paper for patrickbdevaney/MiMo-V2.6-Flash-REAP50

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

Paper • 2510.13999 • Published Oct 15, 2025 • 23