code-llm-1b-sft — the instruction-tuned 1.1B Python model (mix42 arm, val-best)

Supervised fine-tune of the 1.1B base model in ../base. Same architecture, same tokenizer, same 1024-token context; the difference is that this checkpoint answers a task stated in prose or as a signature (### Task ... ### Code) instead of merely continuing code.

This folder is the SFT release, and it is the best of the 1B fine-tuning arms on the combined benchmark: 97/110 on the instruction column and 91/110 on the signature column (human 0/1/2 grading, 11 tasks x 5 seeds). The base scores 83/110 and 61/110 on the same ruler.

Architecture


Parameters	1,095,845,888 (bf16), 143 tensors
Layers	20
d_model / heads	2048 / 16 (head_dim 128)
FFN	SwiGLU, d_ff 5120 (gate / up / down)
Position	RoPE, theta 500000
Norm	RMSNorm, learned `scale`, eps 1e-6 (pre-norm)
Attention	causal SDPA, fused QKV projection, no biases anywhere
Embeddings	untied (`embed` + `lm_head`)
Vocabulary	32,000 byte-level BPE (inherited from pretraining, not retrained)
Context	1024 tokens
Precision	bf16 (released file)

Parameter count is the sum of the 143 released tensors and equals the count of the instantiated CodeLLM class, not the (slightly wrong) closed form in ModelConfig.num_params.

Fine-tuning recipe

Init: checkpoints_wsm_1b/merged (the pretraining merged weights in ../base).
Data: data_mix_gold — the gold instruction set mixed with signature-style examples at 5:1, 91.78M tokens in total; the loss-mask covers only the answer span.
Optimisation: 2,801 steps at lr 2e-5, batch 4 x grad-accum 8 (32 sequences of 1024 tokens per step), seed 42, weight decay and warmup as in sft_train.py.
Selection: this file is the validation-loss-best checkpoint (SFT val loss 0.6519 at step 2,000 of 2,801) — not the final step. A pre-training val batch was scored alongside as an anti-forgetting gate; it did cross its +0.05 red line mid-run, which is the other reason the released file is a mid-run checkpoint rather than the last one.
Hardware: a single RTX 5090, 32 GB.

Results (human-graded, 0/1/2 per generation, 11 tasks x 5 seeds, maximum 110 per column)

model	instruction column	signature column
base (`../base`)	83/110	61/110
this checkpoint (mix42 best_val)	97/110	91/110
mix42, final step (not released)	98/110	85/110
gold-only arm, best_val	97/110	67/110

The two columns use the same tasks, seeds and generation settings; only the prompt and the judging entry point differ. The mix42 arm does not significantly beat the gold-only arm on the instruction column (+0/+1 over five seeds, inside seed noise ~4 points), but it is clearly better on signatures (+24) — and its training set is 1.47x larger, so the difference is a mix-plus-volume effect, not a clean single-variable result.

Known limits (read before trusting a number)

mergesort never passes (0/5 on both columns); binary_search is 0/5 when prompted by signature and usually returns None/True/False instead of -1 on failure — graded 1 for recognisable structure, not correctness. two_sum is strong by signature (5/5) and weaker in prose.
Missing imports: 18 of 110 generations in the signature column reference math/re/functools without importing them; they are graded 1 (structure right, would raise NameError). Scoring them 2 would move the two mix42 variants from 176 to 194 out of 220 — no ordering against the other arms changes.
Single seed for this release decision (SFT seed 42), five generation seeds for grading; per-arm spread across seeds is about 4 points, so treat gaps under ~6 points as noise.
Template collapse: generation is noticeably homogeneous — fibonacci produces one identical implementation across all five seeds — consistent with the 5:1 signature-data ratio.
Context is 1024 tokens, and the model truncates rather than refusing: prompts longer than that are cut, not summarised.
Trained only on code plus synthetic instructions; no personal data. Outputs may reproduce patterns from the training corpus despite copyright filtering.

Files

File	What
`model.safetensors`	bf16 weights, 143 tensors, 2,191,706,272 bytes — verified bit-identical to `fine-tuning/ckpt_1b_mix42_merged/best_val/checkpoint.pt` after the fp32→bf16 cast
`config.json`	architecture config taken from the checkpoint itself, plus `torch_dtype: bf16`
`tokenizer.json`	byte-level BPE, 32,000 tokens (same file as `../base`)
`SHA256SUMS.txt`	`9ecc6e2d99e81ff0df1655fd714c26e76a556f8b18aa87572adecf9cd0c4f00f model.safetensors`
`arch_facts.json`	provenance: source checkpoint, step, token count, source config

Verification artifacts (the hash, key/shape, bitwise-equality, strict-load and CPU-forward logs) stay in the training workspace and are not part of this release; SHA256SUMS.txt is what lets anyone re-check the weights independently.

Loading requires the model class from the training repository (sft_train.py, class CodeLLM, ModelConfig(**config.json)). This is not a transformers architecture and has no AutoModel mapping. RMSNorm outputs fp32 while the weights are bf16, so run inference under torch.autocast("cpu", dtype=torch.bfloat16) or in fp32 — a bare bf16 CPU forward raises a dtype error inside nn.Linear (training always ran under autocast and never hit it).

Prompt template used in evaluation:

### Task
<instruction>

### Code