Qwen3.8-27B-i1-IQ4_KS_KT-GGUF

This quantization was created to allow the entire model to fit into the memory of an NVIDIA graphics card with 16GB of VRAM.

Importan - please use "export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1"

When GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is defined, CUDA allocations go through cudaMallocManaged (ggml-cuda.cu:181), handing over memory management to the NVIDIA card's hardware MMU:

Eliminating fragmentation (4 KB / 2 MB granularity vs. VMM Pool):

The standard VMM allocator (cuMemCreate) in llama.cpp enforces large, rigid block allocations and throws a hard OOM if there is a lack of contiguous memory space on the card.

cudaMallocManaged operates on very fine-grained physical pages (4 KB / 2 MB). This allows the NVIDIA driver to stitch together small, free VRAM fragments without throwing an allocation error.

This model has the token_emd.weight tensor bumped up to q8_0, which increases its download size but makes no difference in VRAM. By default, the token_emd.weight tensor is loaded entirely into system RAM. Upgrading it to q8_0 is therefore, in practice, a free upgrade to the model's quality, for example in interpreting non-English languages.

This repository contains GGUF format weights for the Qwen3.8-27B model, quantized using the ik_llama.cpp project.

This model was specifically created to run on consumer GPUs with 16GB VRAM. By utilizing q4_0 KV cache quantization, it allows pushing the context length up to 110k tokens.

Note: This model is designed exclusively for nVidia GPUs and is based on the advanced KS and KT quants developed by ikawrakow from the ik_llama.cpp repository.

---

Quantization Details & Imatrix File

• Quantization Base: IQ_KS,IQ_KT Quants (ik_llama.cpp).
• Imatrix File Used: Mix of mradermacher and ubergarm imatrix files.
• Other Tested Imatrix Files:
  • bartowski – yielded significantly worse results.
  • ubergarm – yielded comparable results.
  • If you find or generate a better Imatrix file, please let me know in the Community tab!

GPU Only full 16GB VRAM the configuration below will give you a 105k context (110k is the max) (q4_0/q4_0). Decode speed starts at 45 t/s and falls to 25 t/s at the end, with a prefill speed above 1300 t/s (RTX 5070 Ti).

llama-server" \
        -m "$MODEL_PATH" \
        -a Qwen3.8-27B \
        --ctx-size 105000 \
        --n-gpu-layers 99 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --spec-type ngram-mod:n_max=2 \
        --batch-size 512 \
        --ubatch-size 128 \
        --flash-attn on \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format deepseek \
        --reasoning-budget 32000 \
        -t 8 \
        -tb 8 \
        --parallel 1 \
        --metrics \
        --merge-qkv \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
        --defrag-thold 0.4 \
        --jinja \
        --cont-batching \
        --temp 1.0 \
        --top-k 20 \
        --min-p 0.00 \
        --top-p 0.95 \
        --presence-penalty 0.0 \
        --repeat-last-n 512 \
        --repeat-penalty 1.0

Perplexity Comparison: mradermacher vs. cHunter789

I compared the model by mradermacher with the model by cHunter789 using the War and Peace text file as a dataset.

Test dataset download:

wget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt

---

1. Model: IQ4_XS (mradermacher)

Source: mradermacher/Qwen3.8-27B-i1-GGUF

Command

./llama-perplexity \
    -m Qwen3.8-27B.i1-IQ4_XS.gguf \
    -f pg19.txt \
    -c 65536 \
    --chunks 32 \
    -ngl 99 \
    -ctk q4_0 \
    -ctv q4_0 \
    -khad \
    -vhad \
    --merge-qkv \
    -fa 1 \
    -b 512 \
    -ub 128 \
    -t 8

Output

perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 77.21 seconds per pass - ETA 15.43 minutes
[1]7.7001,[2]8.0855,[3]8.2723,[4]8.3528,[5]8.5027,[6]8.3485,[7]8.3948,[8]8.3789,[9]8.4068,[10]8.4465,[11]8.4932,[12]8.2955,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2955 +/- 0.03221

llama_print_timings:        load time =    2522.03 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  750310.64 ms / 786432 tokens (    0.95 ms per token,  1048.14 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time = 1360215.26 ms / 786433 tokens

---

2. Model: IQ4_KT (cHunter789)

Source: cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF

Command

./llama-perplexity \
    -m Qwen3.8-27B.i1-IQ4_KT-attn_qkv-IQ4_KS.gguf \
    -f pg19.txt \
    -c 65536 \
    --chunks 32 \
    -ngl 99 \
    -ctk q4_0 \
    -ctv q4_0 \
    -khad \
    -vhad \
    --merge-qkv \
    -fa 1 \
    -b 512 \
    -ub 128 \
    -t 8

Output

perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 84.14 seconds per pass - ETA 16.82 minutes
[1]7.6895,[2]8.0682,[3]8.2542,[4]8.3333,[5]8.4853,[6]8.3416,[7]8.3890,[8]8.3734,[9]8.3999,[10]8.4386,[11]8.4857,[12]8.2890,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2890 +/- 0.03224

llama_print_timings:        load time =    9369.56 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  783243.61 ms / 786432 tokens (    1.00 ms per token,  1004.07 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time = 2323646.53 ms / 786433 tokens