avlp12/Kimi-K3-Alis-MLX-Dynamic-2.10bpw

Kimi-K3-Alis-MLX-Dynamic — 2.10 bpw (v3)

Kimi K3 (2.8T-parameter MoE, multimodal) quantized to 737.1 GB / 2.096 bpw for Apple silicon (MLX). This is the v3 build: a ternary-codebook rebuild of this repository that is 211 GB smaller than v2 (948.5 GB / 2.71 bpw) while being closer to the teacher on every measured metric. The previous v2 build is preserved unchanged on branch v2-2.71bpw.

What's in the build

Component	Treatment
Routed experts (low band)	1.5625 bpw ternary codebook (IQ1_S-style semantics, native MLX kernels), Hessian-weighted encoding
Routed experts (protected band)	14,000 of 82,432 expert-instances promoted to native MXFP4 (4.25 bpw) — selected by blended Hessian×activation-mass score with depth-aware, clamp-aware waterfilling
Attention / dense / shared experts	6-bit, group size 64
Vision tower (MoonViT-3d)	bf16, unquantized — multimodal path preserved
Post-quant training	Layer-wise DWQ against the mxfp4-QAT teacher (fp32 master weights, bias-corrected Adam, monotone rollback; all 8 depth blocks converged, val-KL −12…−37% per block)

Average 2.096 bpw over 2.8T parameters; 96 weight files; model.safetensors.index.json covers 7,675 tensors.

Quality (measured, not estimated)

Identical-window protocol against the original mxfp4-QAT teacher (48 windows of 2048 tokens: 24 wikitext-2-test + 24 Korean Wikipedia; teacher logits cached once, fp32 evaluation):

Build	Size	bpw	wikitext PPL	Korean PPL	KL(teacher‖build)	top-1 flip
teacher (mxfp4-QAT)	1,561 GB	≈4.25	1.578	3.114	—	—
v2 (2.71 bpw, retired)	948.5 GB	2.71	2.879	3.341	0.349 nats	17.70%
v3 (this build)	737.1 GB	2.096	2.078	3.311	0.2253 nats	13.77%

The 2.71 bpw (v2) build is retired and no longer distributed. Its numbers stay in the table below purely as the historical baseline this build is measured against — the v2-2.71bpw branch has been removed. v3 (this build) is smaller and better on every metric measured, so there is no configuration in which v2 was the right choice.

Full-set perplexity (all windows of each corpus, same tokenization): wikitext-2 test 142×2048 tokens → 2.2316 (v2: 2.744, −18.7%); Korean 92×2048 → 3.2266 (v2: 3.321, −2.8%). KL split for v3: wikitext 0.327 / Korean 0.124 nats.

Cross-quant reference (measured, but read the caveat): unsloth's UD-IQ2_XXS GGUF (711 GB) evaluated with llama-perplexity (unsloth's kimi-k3-fullsize-vision branch, CPU) on the same wikitext raw text, -c 2048 --chunks 24, gives PPL 1.7064 ± 0.0128. This number is not directly comparable to the table above: llama-perplexity inserts BOS per chunk and handles window boundaries differently from our teacher-logit harness (same text, different protocol — the two harnesses disagree systematically, and we could not run either build through the other's harness). Within a single harness the ordering is measured only for our builds (v3 strictly better than v2 everywhere). The KL-vs-teacher and flip numbers above remain the strongest like-for-like quality evidence for this build; a same-harness GGUF comparison would require logits export from llama.cpp and is left open. (Korean H2H was skipped by design.)

Install & serve (web chat)

Hardware floor: two Apple-silicon Macs with 512 GB unified memory each (per-box resident ≈370 GB weights + ≈27 GB fusion/packing overhead), linked by Thunderbolt 5 or 10G+ Ethernet. Single-box is arithmetically impossible at this size; a 1-box-class variant is under research.

# 1. Both boxes: deps + this repo's runtime (the *.py files and chat_ui.html here)
pip install "mlx>=0.31" "mlx-lm>=0.31" numpy huggingface_hub
huggingface-cli download avlp12/Kimi-K3-Alis-MLX-Dynamic-2.10bpw --local-dir k3-2.10bpw

# 2. Box 1: split the 96 shards into two expert-parallel halves (~369 GB each),
#    then move the rank-1 half to box 2 (fast local link recommended)
python3 slice_rank_half.py --src k3-2.10bpw --rank 0 --out r0
python3 slice_rank_half.py --src k3-2.10bpw --rank 1 --out r1   # then scp/rsync r1 to box 2

# 3. Launch (from box 1; starts rank 1 on box 2 over ssh, serves web chat on :8010)
BOX1_IP=10.0.0.1 BOX2_IP=10.0.0.2 BOX2_SSH=user@10.0.0.2 \
BUILD_R0=$PWD/r0 BUILD_R1='~/r1' bash launch_serve.sh

Open http://<box1>:8010 — markdown-rendered chat with a thinking-channel panel, session KV reuse (delta prefill), and a stop button. /api/chat accepts {messages, session, max_tokens, temperature, thinking_effort} as SSE.

What the runtime includes (all validated bit-exact or ULP-bounded, each behind a killswitch): the 1.5625 bpw ternary-codebook Metal kernels (k3_cbq.py, row-parallel decode variant), KDA projection packing + fused glue kernels (k3_fuse.py), fused MoE router (top-16-of-896 in one dispatch), SiTU-at-store expert kernels, expert-parallel harness with lazy-graph collectives (k3_ep.py), and two-rank synchronized sampling (no per-token broadcast; 32-round tripwire). Env switches: K3_KDA_FUSE/K3_KDA_GLUE/K3_MOE_ROUTE/K3_SHARED_FUSE/ K3_CBQ_NO_V4/K3_CBQ_NO_SITU/K3_SYNC_SAMPLE (all default-on), K3_MLA_ABSORB=1 (required).

Note to AI agents

If you are a coding agent (Claude Code, Codex, etc.) asked to deploy this: the fast path is (1) verify both boxes see each other over ssh without a password and agree on IPs; (2) run the three numbered blocks above exactly — the shard split must finish on both boxes before launch; (3) health-check curl -s http://<box1>:8010/ returns the chat HTML, then POST a short /api/chat request with "temperature":0 and confirm tokens stream; (4) expect ≈7 min load per boot; ≈8.8 tok/s decode with the default K3_ATTN_TP=1 K3_DENSE_TP=1 (up to ≈10.5 tok/s on code with opt-in speculative decoding, see below) (7.0 with TP off, see below); if rank 1 dies, kill both k3_serve.py processes with SIGTERM (never kill -9 — wired-memory leak risk) and relaunch. Do not raise MLX_MAX_OPS_PER_BUFFER/MLX_MAX_MB_PER_BUFFER — measured regression. The port in the hostfile must be free on BOTH boxes (TIME_WAIT collisions look like hangs).

Why this matters

This is, to our knowledge, the first time a 2.8-trillion-parameter multimodal MoE runs as a private, usable chat service on hardware a person can put on a desk: two consumer Macs, no datacenter, no API, ≈7-8 tok/s with vision intact. Three things had to be true at once:

• 2.10 bits per weight with receipts. A ternary codebook (IQ1_S semantics) for 82k experts, mxfp4 promotion for the sensitive tier, layerwise DWQ distillation with an fp32 master — and the quality ledger published next to it (KL 0.2253 vs the fp teacher, token-flip 13.77%, PPL within striking distance of a build 211 GB larger). Compression claims are cheap; side-by-side deltas are not.
• The decode path is the product. Stock kernels served this build at 2.3 tok/s; it now runs ≈8.8 plain — 10.0–10.6 on code with opt-in speculative decoding. Every step of that 3.8–4.6× — codebook kernels, projection packing, glue fusion, tensor-parallel attention/dense, MMA prefill, fused routing, synchronized sampling — is reproduced in this repo and documented (with the failed attempts) in alis-dwq, including the three MLX rounding semantics you must match to fuse kernels bit-exactly against eager ops.
• Frontier weights, local physics. Open-weight frontier models keep arriving; unified-memory machines keep growing. The gap between "the weights exist" and "you can actually talk to them at home" is exactly the kernel-and-harness work this repo demonstrates end to end.

Decode-speed details

Same expert-parallel deployment as v2: two 512 GB Apple-silicon Macs over Thunderbolt 5 (per-box resident ≈368 GB — more headroom than v2's ≈475 GB). The custom loader/EP harness and the 1.5625 bpw codebook kernels are required (see the repo code files; K3_MLA_ABSORB=1 recommended).

Measured decode (93 layers, 2-box EP): ≈7.0 tok/s baseline · 8.8 tok/s with attention + shared-expert tensor-parallel — the default serving configuration (2026-08-06/09 campaign: EP dummy-row elision +8%, single-dispatch GLU+SiTU fusion +2%, MLX 0.32 jaccl RDMA collectives +5%, attention head-TP K3_ATTN_TP=1 +12% — each step transcript- or teacher-anchored). TP quality is certified two ways: single-box head-shard numerical parity (rel_max ≈5e-3 on both KDA and MLA layers = bf16 rounding scale; the partial sums are accumulated in fp32 before the all_sum), and byte-identical greedy transcripts vs the non-TP configuration (3 prompts × 96 tokens, reasoning traces included). On top of attention TP, the shared (always-active) experts are also sharded (−4.9 GB weight reads per token per rank, gate/up column-split + down row-split); their partial sums ride the existing per-layer expert all_sum as a concatenated payload, so shared-expert TP adds zero extra collectives. Certified by a 4-window teacher-anchored KL check — 0.2303/0.3626/0.1112/0.1170 nats vs the non-TP baseline 0.2313/0.3637/0.1130/0.1169, i.e. within noise — plus single-box numerical parity on the sliced-then-packed fused path (rel_max ≈1e-3). At 8.8 tok/s this build is now ahead of the 211 GB larger v2 build (5.7 tok/s) outright, and decode holds 8.8 tok/s even at 2k-token context depth.

TP notes: K3_ATTN_TP=1 shards the replicated q/k/v/gate/o attention projections across the two ranks (−16.6 GB weight reads per token per rank) at the cost of one extra per-layer partial-sum all_sum. The long-prompt stall reported here earlier is fixed: the trigger was the size of a single forward graph containing per-layer TP collectives (T ≳ 1.5k wedges the scheduler on both jaccl and ring backends; chunking only the all_sum payloads does not help), so under TP the server now chunks prefill into 256-token forwards automatically (K3_PREFILL_CHUNK, override if you must) — verified end to end with a 2k-token prompt. The chunking cost was then erased by a new prefill kernel: v7, an MMA fused-codebook GEMM (simdgroup_matrix tiles with the ternary-codebook dequant fused into the threadgroup loader — no dequant round-trip to device memory, tile runs aligned to expert boundaries, tile height auto-selected by routed-run length). Measured on the expert GEMMs: 2.1× vs the prior token-parallel kernel at 512-token chunks, 3.6× unchunked, 2.65× vs dequant-then-GEMM expert-major. End to end at 2k tokens: prefill 24.8 s (80 tok/s) under full TP — faster than the old non-TP path (33.6 s) — with decode 8.8 tok/s held at that depth. Short chat-turn prompts keep the token-parallel kernel (tiny per-expert runs starve MMA tiles below ~3k routed rows) and are byte-identical. The v7 configuration passes the same 4-window teacher-anchored KL gate (0.2307/0.3634/0.1125/0.1167 vs baseline 0.2313/0.3637/0.1130/0.1169).

Getting to this speed took a decode-specialized codebook kernel (simdgroup split-K + vectorized loads, 3.8× on the expert GLU vs. the first working kernel), keeping collectives in the lazy graph instead of hard-syncing per layer, packing the six same-input KDA projections into one QMM (bit-exact row-concat), fusing the KDA glue ops (conv/silu/rms-norm/gate chains) into two custom Metal kernels that reproduce MLX's eager rounding semantics exactly, fusing the MoE router chain (top-16-of-896 + hi-tier selection) into a single kernel, and inlining the SiTU activation into the expert kernels' store sites (all bit-exact). Both ranks sample locally from bit-identical logits (2-rank ring reductions are commutative-only), removing the per-token broadcast. The residual gap is the ternary codebook's extra dequant work. The v2-2.71bpw branch remains marginally faster if footprint is no object.

Speculative decoding (opt-in): 10.0–10.6 tok/s on code — first double-digit decode on this build. A 2.25B DFlash-style block drafter (tap layers [7,23,51,67,83], k=5), aligned in two on-policy rounds (doc-seeded then chat-seeded self-generation of the target; offline accept-length 3.41 → 4.52 doc / 2.25 → 3.58 chat), drives self-speculative decoding with an adaptive accept-gate: per-request rolling accept (fast 8-window < 2.4, fine 16-window < 2.8) falls the request back to plain decode whenever drafting loses — so worst case ≈ plain. Measured live (2× M3 Ultra, TP + jaccl): code generation 10.0–10.6 tok/s (+15–21% vs 8.8 plain), short answers at par; Korean long-form currently nets negative (drafter conditioned on non-TP tap hiddens vs TP serving — known gap, next campaign) and the gate reverts it to plain. Enable with K3_SPEC=2 K3_SPEC_K=5 and the drafter in dspark-drafter-r4/; acceptance is sampler-equality, so outputs follow the target's own distribution. Full round economics and lessons: alis-dwq repo, K3 Addendum 11.

Language-specialized variant (Korean, speed-neutral): the mxfp4-preserved (hi-tier) experts in this build were selected by mixed-corpus routing mass. Re-profiling routing on Korean text shows the mixed-corpus hi-set covers only ~15% of Korean routing mass and overlaps the Korean-oracle top-set by ~18% — nearly orthogonal. Re-tiering (swapping which experts are mxfp4 vs ternary, same count per layer so per-token read bytes and decode speed are unchanged) yields, on the 4-window teacher-anchored gate: Korean KL 0.0905 / 0.0824 (vs 0.1130 / 0.1169 baseline, −20% / −30%), Korean PPL 3.311 → 3.181 (−3.9%), wikitext PPL 1.740 → 1.743 (+0.2%, within noise), decode 8.8 tok/s unchanged. A free quality lever for a target language: same footprint, same speed, measurably better Korean. Method generalizes to any language/domain (route-profile → oracle re-tier). Build recipe in the repo code.

Multimodal verified on this build: 414 image tokens (gradient test image → MoonViT-3d tower from this repo's vision.safetensors) injected and run through all 93 quantized layers across both boxes. Beyond finite activations (rms 0.60), the model generated HTML/CSS reproducing the input image (<div class="gradient"> + background: linear-…) — the same image-to-code behavior the v2 build demonstrated. Vision tower is bf16 passthrough, identical treatment to v2.

Single-box use is not possible at this size; a 1-box-class variant is under research.

Provenance & honesty notes

• The promotion budget (14,000) was fixed before final evaluation; no eval-set tuning of the allocation.
• DWQ trained only quantization parameters (codebook scales/duals, dense scales) — the router was not trained (value-path gradients cannot see top-k flips; ecosystem consensus agrees).
• Per-block learning rate: 3e-6 with bias-corrected Adam; one block (b1) converged at 1.5e-6 after an automatic revert-and-halve. Full recipe and the divergence post-mortem (bf16 master / bias-correction pitfalls) are published in alis-dwq.
• Evaluation harness cross-checks: offline logits→PPL reproduction matches in-loop accumulation to 4 decimals; raw-text→token oracle equality verified for the H2H text dumps.

License

Kimi K3 License: derivative distribution is permitted with notice. Commercial-scale attribution requirements apply for operations exceeding 100M monthly active users or $20M monthly revenue. See the license link in the header.