vmarcelo/ThinkingCap-Qwen3.8-27B-MIX_GGUF

⚠️ Superseded — use Swift 1.5 instead

This quant has been superseded by ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF, the official GGUF release of Swift 1.5. I recommend using that release instead — this page remains available as an archive.

---

ThinkingCap-Qwen3.8-27B: IQ4_XS-MIX

A custom mixed-tensor IQ4 quantization of bottlecapai/ThinkingCap-Qwen3.8-27B, the ThinkingCap fine-tune of Qwen3.8-27B that keeps the base model's answer quality while reasoning in far fewer tokens.

It comes in at 13.40 GB and 3.924 BPW, which is 2.08 GB smaller than the published IQ4_XS (15.48 GB), and it keeps the MTP head embedded so speculative decoding needs no second file.

Built with llama.cpp 58367713a from BottleCap's own f16 GGUF, using a purpose-built importance matrix and per-tensor overrides. general.file_type is 30 (MOSTLY_IQ4_XS), matching the name.

Files

File	Size	BPW	Notes
ThinkingCap-Qwen3.8-27B-IQ4_XS-MIX-3.92bpw.gguf	13.40 GB	3.924	MTP embedded
thinkingcap_iq4_types.txt	9 KB	n/a	Full per-tensor recipe
thinkingcap-imatrix.gguf	14 MB	n/a	Importance matrix used (provenance)
config.json · generation_config.json	n/a	n/a	Arch metadata for LM Studio / HF Hub

Vision needs an mmproj file, so use BottleCap's mmproj-ThinkingCap-Qwen3.8-27B-f16.gguf.

Recipe

The file averages fewer bits than a uniform IQ4_XS (3.924 vs 4.25 BPW), because it spends more on the tensors that matter and less on the feed-forward. The feed-forward is 62.6% of all parameters, which makes it the only place with enough mass to fund a smaller file.

FFN allocation by depth:

Zone	Blocks	gate / up / down
EARLY	0–15	IQ2_XXS / IQ2_XXS / IQ3_XXS (cut deepest)
MID	16–51	IQ3_XXS / IQ3_XXS / IQ4_XS
BACK	52–62	IQ4_XS / IQ4_XS / Q5_K
LAST	63	Q4_K / Q6_K / Q6_K

The following tensors are protected, because they are cheap and because the sensitivity research flags them as fragile in hybrid GatedDeltaNet models: ssm_alpha/ssm_beta at Q8_0; linear-attention attn_gate at Q5_K with ssm_out/attn_qkv at IQ4_XS; full-attention attn_k/attn_v at Q8_0, attn_output at Q5_K and attn_q at IQ4_XS; output.weight at Q5_K; token_embd at Q3_K; and the MTP block at Q6_K.

Why there is no front-end lift: early blocks are among the cheapest in this architecture, so they take the deepest cuts rather than extra bits. Only the last twelve blocks are lifted.

Usage

llama-server -m ThinkingCap-Qwen3.8-27B-IQ4_XS-MIX-3.92bpw.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.8-27B-f16.gguf \
  -ngl 999 -c 65536 -b 2048 -ub 512 -t 12 -np 1 \
  --cont-batching --jinja --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --host 0.0.0.0 --port 8080

The MTP head is embedded, so --spec-type draft-mtp alone enables speculative decoding and there is no --model-draft file to pass. Dropping that flag frees about 1.2 GiB of VRAM and roughly doubles the usable context, at the cost of around 27% of generation speed (see below).

Use BottleCap's recommended sampling settings from the main model card. Note that greedy decoding can loop, so keep temperature at the recommended value.

Context guidance (16 GB VRAM)

Only 16 of 65 blocks are full attention, so the KV cache is small: 64 KiB/token at F16, 34 KiB at Q8_0, 21 KiB at Q5_0/Q4_1. Every figure below is measured on an RX 9070 XT (RADV/Vulkan) with this exact file and a ~22.5K-token prompt.

MTP is the dominant variable here, not the KV type. The speculative-decoding draft carries its own KV cache and verification buffers, costing about 1.2 GiB, which is roughly 27K worth of Q8_0 context. Turning MTP off buys around 50% more context for about 27% of generation speed, and it slightly raises prompt throughput.

KV	MTP	ctx	GTT spill	Prompt	Generation	verdict
Q8_0/Q8_0	on	32K	254 MiB	973.6 t/s	44.4 t/s	clean
Q8_0/Q8_0	on	48K	283 MiB	966.5 t/s	44.1 t/s	clean
Q8_0/Q8_0	on	64K	315 MiB	965.5 t/s	44.3 t/s	clean, max spill-free
Q8_0/Q8_0	on	80K	914 MiB	874.8 t/s	34.3 t/s	mild spill
Q8_0/Q8_0	on	96K	1,624 MiB	855.4 t/s	24.0 t/s	heavy spill, avoid
Q8_0/Q8_0	off	96K	192 MiB	1017.6 t/s	32.4 t/s	clean
Q5_0/Q4_1	on	64K	318 MiB	864.2 t/s	47.0 t/s	clean
Q5_0/Q4_1	on	96K	382 MiB	864.7 t/s	47.9 t/s	clean, best generation
Q5_0/Q4_1	on	128K	1,169 MiB	809.1 t/s	37.4 t/s	mild spill
Q5_0/Q4_1	on	160K	2,016 MiB	560.1 t/s	26.6 t/s	heavy spill, avoid
Q5_0/Q4_1	off	160K	256 MiB	910.5 t/s	35.0 t/s	clean

Reading the spill column. GTT is system memory the driver fell back to when VRAM ran out. A ~1 GB spill costs 6–9% of prompt throughput and 22% of generation, so it is slower but still a legitimate operating point if you want the extra context. Past ~1.6 GB it collapses, losing 35–46%. Spill is silent, in that it still exits cleanly, so check GTT rather than assuming a configuration fits.

Recommended picks:

• Fastest, fully resident: Q5_0/Q4_1 + MTP at 80–96K, which gives the best generation (47.9 t/s).
• Long context with lossless KV: Q8_0/Q8_0 + MTP off at 80–96K, which gives the best prompt throughput (1017.6 t/s) with uncompressed attention.
• Accepting a mild spill: Q5_0/Q4_1 + MTP at 128K, a third more context than the spill-free ceiling in exchange for 22% of generation speed.
• Maximum context: Q5_0/Q4_1 + MTP off at 160K, still spill-free.

Prefer Q8_0 KV where the context allows. It is consistently about 10% faster on prompt processing than Q5_0/Q4_1 in every configuration measured here, with generation within noise, so compressed KV is a capacity lever rather than a speed one.

Why Q5_0/Q4_1 instead of Q4_0/Q4_0?

If the cache has to be compressed, Q5_0/Q4_1 is the better trade. Measured against a bf16 cache on a 27B model of this exact shape:

K / V	% of bf16 KV	99.9% precision	worst-case KLD
Q5_0/Q5_0	34.4%	92.70%	0.099
Q5_0/Q4_1	32.8%	92.65%	0.100
Q4_0/Q4_0	28.1%	89.84%	0.130

Q4_0/Q4_0 is 16.7% smaller (18 vs 21 KiB/token), but its worst 0.1% of positions diverge 32% more, which costs about 2.8 points of tail precision. Perplexity cannot see this difference. It shows up as broken JSON keys and failed tool calls, which is the one failure an agentic session cannot absorb. Three rules fall out of the same data:

• Crossing V below q5 costs a full point of tail precision, while steps within q5 cost about half a point. That is why Q5_0/Q4_1 lands within 0.05 points of Q5_0/Q5_0 at less size.
• Spend bits on K first. At the same footprint, Q5_0/Q4_0 beats symmetric Q4_1.
• Symmetric Q8_0/Q8_0 is a validation mode rather than a practical default, because dropping V to Q5_1 costs only about 0.4 points of tail precision while shrinking the cache by 7.8% (34 down to 28 KiB/token here, saving 576 MiB at 96K).

Source: KV Cache Quantization Benchmarks for Long Context (Anbeeld; 99.9% KLD against a bf16 cache, Qwen 3.6 27B on an RTX 3090).

How it was built

Quantized from BottleCap's f16 GGUF directly, with no re-conversion, so the MTP head, tokenizer and chat template are exactly as published. The importance matrix was computed on a 13.4 GB IQ4 copy of the model rather than the 54.7 GB f16, which keeps it fully resident in VRAM (583 chunks at ctx 512, ~3.2 s/pass). Running it on the f16 would spill out of page cache and re-read tens of GB per chunk.

Credits

• Model: bottlecapai/ThinkingCap-Qwen3.8-27B by BottleCap AI, the weights this was built from.
• Base model: Qwen/Qwen3.8-27B by the Qwen team.
• Calibration corpus: bartowski/Qwen3.8-27B-GGUF (calibration-v6, chat-template rendered).
• Toolchain: llama.cpp at 58367713a.
• Quantized by vmarcelo.

License

The base model is Apache-2.0. The ThinkingCap contribution is released under the PolyForm Small Business License 1.0.0, and these GGUF conversions inherit those terms. See the source repository for the full licence text.