Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller

A custom hybrid quantization of the Qwen3.8-27B base model, specifically designed to fit Multi-Token Prediction (MTP) and long context into a strict 16GB VRAM hardware budget (like consumer RTX 4080 / RTX 5080 cards).

---

How It Works

Instead of compressing the whole model equally, it is split by layer type:

• Attention Layers (Brain & Logic): Kept high at IQ4_XS to protect reasoning, coding, and formatting logic.
• FFN Layers: Compressed down to IQ3_S to trim file size. The trade-off is a minor loss in general knowledge and long-context recall, so if you use the model for something like creative writing, you'd probably be better off just using normal IQ4_XS quants.
• The Result: The model file drops down to ~13.5 GB, leaving enough room for MTP and/or higher context size. The 5080 laptop GPU it was tested on (approximately 5070 desktop equivalent) results in 64k context with MTP at 50 t/s and 128k without MTP at around 30 t/s.

---

How It Was Made

The model was compiled using mainline llama.cpp.

The Compilation Command:

/home/llama.cpp/build/bin/llama-quantize \
  --imatrix Qwen3.8-27B.imatrix.mradermacher-ubergarm.dat \
  --tensor-type ffn_down=iq3_s \
  --tensor-type ffn_up=iq3_s \
  --tensor-type ffn_gate=iq3_s \
  BF16/Qwen3.8-27B-BF16.gguf \
  Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf \
  IQ4_XS \
  64

---

Credits & Acknowledgments

• lemonyins — For the original methodology: https://huggingface.co/lemonyins/Qwen3.6-27B-uncensored-abliterated-i1-IQ4_XS-GGUF-Smaller.
• cHunter789 — For providing the iMatrix data: https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
• Unsloth — For the base bf16 model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
• llama.cpp — the ggml/llama.cpp team for the base quantization framework.