clef-lora

The LoRA adapters inside Cloudflare's Clef models, extracted from their merged weights.

Path	Model	Source	Base	Adapter
/	Clef (27B)	Cloudflare/clef@ed3eed33	Qwen/Qwen3.8-27B@1d4bf0f2	186 linears, layers 40–63
clef-flash/	Clef-flash (9B)	Cloudflare/clef-flash@8b2e5fd1	Qwen/Qwen3.5-9B@c2022362	248 linears, all 32 layers

Both are PEFT LoRAs in bf16, r=256, lora_alpha=256. The Gated DeltaNet in_proj_a/in_proj_b modules use their full width instead (48 for Clef, 32 for Clef-flash), set via rank_pattern/alpha_pattern.

Usage

The adapter replaces the backbone only. The decision head and encoding code come from the Cloudflare repo.

import json, sys, torch
from pathlib import Path
from huggingface_hub import snapshot_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

clef = Path(snapshot_download("Cloudflare/clef", revision="ed3eed331870db2eff4b0db01237128ede8a00ce",
                              allow_patterns=["*.py", "*.json", "*.jinja", "joint_head.safetensors"]))
sys.path.insert(0, str(clef))
import joint_schema_model as jsm

backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.8-27B", revision="1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", dtype=torch.bfloat16, device_map={"": "cuda"})
backbone = PeftModel.from_pretrained(backbone, "moondream/clef-lora")
head = jsm.JointSchemaHead(**json.loads((clef / "joint_head_config.json").read_text()))
head.load_state_dict(load_file(clef / "joint_head.safetensors"))
model = jsm.ClefModel(backbone, head.to("cuda", torch.bfloat16)).eval()
processor = AutoProcessor.from_pretrained(clef)

print(jsm.systemone(model, processor, {"model": "clef", "state": "Checkout is down.",
                                       "questions": {"outage": {"type": "noul", "instructions": "Is a service down?"}}}))

For Clef-flash, use Cloudflare/clef-flash@8b2e5fd1, Qwen/Qwen3.5-9B@c2022362 and subfolder="clef-flash". Install flash-linear-attention and causal-conv1d; without them transformers uses a much slower path.

Extraction

• delta = W_finetuned - W_base per tensor in fp32, with the base cast to the fine-tune's dtype first.
• Only the linears listed above change. Embeddings, lm_head, norms and the vision tower match the base.
• Each delta is exactly rank 256 above the bf16 merge-rounding floor. An exact SVD gives B = U√S, A = √S Vᵀ.
• The rank-256 fit leaves 0.2–4% relative error per tensor, matching the expected rounding noise. extraction.json lists it per module.

Verification

Base + adapter + the release's own head against the released model, on the same requests:

Model	Test set	Answers changed	Max KL
Clef	Decision Index 0.2.1, 6,373 requests (30,950 questions)	0.09%	0.0035
Clef	Vision, 11 benchmarks (2,400 rows)	0.12%	0.0024
Clef-flash	120 synthetic requests (231 questions)	0.43% (1)	0.0059

This is bf16 rounding noise: the same arithmetic as the merged weights, done in a different order.

On an FP8 base

The Clef adapter, unchanged, on Qwen/Qwen3.8-27B-FP8 with FP8 weights and activations emulated in bf16 on H200. Paired on 27,231 Decision Index requests (114,220 gold-scored questions):

• Accuracy: no significant change (net −0.012% of questions; mean −0.05 points per benchmark).
• Calibration: unchanged.
• 1.15% of answers flip, almost all near-ties, in both directions.

License

Apache-2.0, following the source models and their bases.