Qwen3.8-27B-GPTQ-W4A16

Model Overview

• Model Architecture: Qwen3.8-27B
  • Input: Vision-Text
  • Output: Text
• Model Optimizations:
  • Visual Weight quantization: bfloat16
  • MTP Weight quantization: bfloat16
  • language Weight quantization: INT4
  • Activation quantization: FP16
• Model Size
  • Qwen3.8-27B: 55.6G
  • Qwen3.8-27B-FP8: 30.9G
  • Qwen3.8-27B-GPTQ-W4A16: 19.5G
• Release Date: 7/17/2026
• Model Developers: Public

Model Optimizations

This model was obtained by quantizing the weights of Qwen/Qwen3.8-27B to INT4 data type, ready for inference with vLLM.

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

vllm serve zhnagchenchne/Qwen3.8-27B-GPTQ-W4A16 \
  --reasoning-parser qwen3 \
  --max-model-len 262144 \
  --language-model-only

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

Creation

This model was created with llm-compressor by running the code snippet below:

Model Creation Code

import torch
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from datasets import load_dataset
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context
from llmcompressor.modifiers.gptq import GPTQModifier
from transformers import AutoModelForCausalLM, AutoTokenizer

# NOTE: This example requires transformers >= v5

MODEL_ID = "Qwen/Qwen3.8-27B"

# Load model.
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)


# No need to include mtp layers as they are not loaded
# through Qwen3_5MoeForConditionalGeneration
#recipe = QuantizationModifier(
recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
        "re:.*embed_tokens$",
        "re:.*shared_expert_gate$",
        "re:.*linear_attn.in_proj_b",
        "re:.*linear_attn.in_proj_a",
    ],
)
NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 4096

ds = load_dataset(
    "HuggingFaceH4/ultrachat_200k",
    split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.select_columns(["messages"])
ds = ds.shuffle(seed=42)

def preprocess(example):
    return {
        "text": tokenizer.apply_chat_template(
            example["messages"],
            tokenize=False,
        )
    }

ds = ds.map(preprocess)

# Tokenize inputs.
def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )

ds = ds.map(tokenize, remove_columns=ds.column_names)

# Apply quantization.
oneshot(
    model=model,
    recipe=recipe,
    dataset=ds,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-GPTQ-W4A16"
model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)

# MTP layers are excluded from the model through Qwen3_5MoeForConditionalGeneration
# Save them as-is from the original checkpoint into the quantized output.
save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)

Evaluation

The model was evaluated using lm_evaluation_harness for mmlu_pro_plus and gsm8k_platinum_cot_llama text benchmark. The evaluations were conducted using the following commands:

Evaluation Commands

lm_eval

lm_eval --model local-chat-completions \
  --tasks gsm8k_platinum_cot_llama \
  --model_args "model=zhnagchenchne/Qwen3.8-27B-GPTQ-W4A16,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_mmlu_pro.json \
  --seed 0 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=0"



  lm_eval --model local-chat-completions \
  --tasks mmlu_pro_plus \
  --model_args "model=Qwen/Qwen3.8-27B,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_mmlu_pro.json \
  --seed 0 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=0"

Accuracy

Metric	Qwen/Qwen3.8-27B	zhnagchenchne/Qwen3.8-27B-GPTQ-W4A16
gsm8k_platinum_cot_llama	0.9727	0.9752
mmlu_pro_plus	-?-	-?-