PocketAiHub/Qwen3.8-27B-Abliterated-MLX

Qwen3.8-27B Abliterated MLX

An unofficial family of MLX derivatives of Qwen/Qwen3.8-27B. The original model is by Qwen; the MLX conversions, refusal-direction experiment, and validation were performed by PocketAI Model Lab. PocketAiHub identifies the publisher of these derivatives, not the creator of Qwen3.8.

This consolidated repository contains experimental 2-bit AWQ plus 4-bit, 6-bit, 8-bit, and BF16 variants. It is the canonical download location.

Important safety notice

These checkpoints have been modified to suppress learned refusal behavior. They may produce harmful, illegal, offensive, deceptive, or dangerously incorrect content more readily than the upstream instruction model. Abliteration is not truthfulness training, capability improvement, or a safety guarantee. Use these models only where you can independently evaluate and constrain their outputs.

Variants

Variant	Folder	Stored size	Precision layout
MLX 2-bit AWQ (experimental)	2bit/	11,038,999,169 bytes (10.28 GiB)	498 language modules affine Q2/group 32 with AWQ; vision tower BF16
MLX 4-bit	4bit/	16,081,501,642 bytes (14.98 GiB)	498 language modules affine Q4/group 64; vision tower BF16
MLX 6-bit	6bit/	22,804,840,262 bytes (21.24 GiB)	498 language modules affine Q6/group 64; vision tower BF16
MLX 8-bit	8bit/	29,528,178,772 bytes (27.50 GiB)	498 language modules affine Q8/group 64; vision tower BF16
MLX BF16	bf16/	54,740,464,127 bytes (50.98 GiB)	unquantized BF16 reference

All variants share the pinned upstream revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. Quantized variants use MLX affine quantization; 2-bit uses AWQ/group 32 and 4/6/8-bit use group 64. Their vision towers remain in BF16.

Abliteration recipe

A projected harmful-minus-harmless direction was measured from 256 length-matched prompts per class at the assistant-generation boundary.

• Direction source layer: 53
• Destination layers: 24–63
• Scale: 1.0
• Per-input-column norm preservation: enabled
• Modified residual-output matrices: 80

See abliteration-manifest.json for the machine-readable recipe.

Behavioral evaluation

Each variant was evaluated deterministically on 100 harmful prompts and 100 benign controls with batch size 1, thinking disabled, and a 128-token ceiling.

Variant	Harmful explicit refusals	Benign explicit refusals	Final answers present
MLX 2-bit AWQ (experimental)	0/100	0/100	196/200
MLX 4-bit	0/100	0/100	200/200
MLX 6-bit	0/100	0/100	200/200
MLX 8-bit	0/100	0/100	200/200
MLX BF16	0/100	0/100	200/200

The scorer is a transparent phrase-based explicit-refusal detector. Most generations reached the token ceiling, so this is an early-refusal screen rather than a full-completion evaluation. It cannot establish universal compliance or answer quality. Machine-readable results are in benchmarks/validation-summary.json and each variant's validation-summary.json.

Feature validation

The 4-bit, 6-bit, 8-bit, and BF16 variants passed:

• 12/12 deterministic quality checks
• 8/8 native tool-call checks
• text and image smoke tests
• temporal video understanding (red->blue)
• 4K-context retrieval (COBALT-7319)

The 2-bit AWQ build is experimental. It measured 9/12 quality checks, 0/8 tool-call checks, passed vision but not exact-text smoke, missed the synthetic temporal-video check, and found the 4K needle but did not return it exactly. These limitations are preserved in the machine-readable evidence.

4K MLX performance

These are single local runs on an Apple M5 Max with 128 GB unified memory, using mlx==0.32.0, mlx-vlm==0.6.8, batch size 1, temperature 0, seed 0, and thinking disabled. Each prompt contained 4,105 tokens.

Variant	Prompt prefill	Generation	End-to-end	Peak MLX memory
MLX 2-bit AWQ (experimental)	411.9 tok/s	25.2 tok/s	10.62 s	16.00 GB
MLX 4-bit	641.7 tok/s	33.2 tok/s	6.68 s	21.80 GB
MLX 6-bit	548.2 tok/s	24.7 tok/s	7.87 s	29.54 GB
MLX 8-bit	579.1 tok/s	18.7 tok/s	7.58 s	37.27 GB
MLX BF16	513.9 tok/s	9.0 tok/s	9.02 s	58.29 GB

These measurements are not cross-machine performance guarantees.

Download and load

python -m pip install "mlx==0.32.0" "mlx-vlm==0.6.8"

Download only the desired variant:

from pathlib import Path

from huggingface_hub import snapshot_download
from mlx_vlm import generate, load
from mlx_vlm.prompt_utils import apply_chat_template

repo_id = "PocketAiHub/Qwen3.8-27B-Abliterated-MLX"
variant = "4bit"  # "2bit", "4bit", "6bit", "8bit", or "bf16"

snapshot = Path(snapshot_download(repo_id, allow_patterns=[f"{variant}/*"]))
model, processor = load(str(snapshot / variant))

prompt = apply_chat_template(
    processor,
    model.config,
    "Explain why seasons occur.",
    num_images=0,
    enable_thinking=False,
)
result = generate(
    model,
    processor,
    prompt,
    max_tokens=256,
    temperature=0.0,
    enable_thinking=False,
)
print(result.text)

Image and video inputs use the normal mlx_vlm.generate media arguments.

License and attribution

The base model is Apache 2.0 licensed. See LICENSE and the official Qwen model card.