Slayer1B 1.03 — training checkpoints
Full trainer checkpoints of the run that produces Slayer1B 1.03: continued training of
Slayer1B 1.02 from its final checkpoint
(ckpt_5720.pt, sha256 6b2a9b11…, constant learning
rate). Each file holds the model weights and the optimizer state.
Status (27 October 2026): training finished at step 7150 (05:14:58 UTC, train loss 2.3955 at step 7150).
All checkpoints of this run, up to the final ckpt_7150.pt, are in this repository; their SHA-256 are listed below.
| Start | 1.02 ckpt_5720.pt (step 5720, constant lr 6e-4) |
| Steps | 7150 in total × 1 048 576 tokens = 7 497 318 400 tokens (+1430 steps = +1 499 463 680 tokens over 1.02) |
| Schedule | constant lr 6e-4 to the end, no final decay — the decay is deferred to the end of the whole Slayer1B line (pre-training continues on the whole prepared corpus) |
| Data | continues the pack-03 stream of SlayerLab/slayer-pl-8x3b (revision d50df04a) used by 1.02 from step 5390: steps 5721–7150 |
| Hardware | steps 5721–6750: 4 × NVIDIA RTX 4090 (24 GB); steps 6751–7150: 4 × NVIDIA L40S (48 GB), resumed from ckpt_6750.pt (sha256 970518a1…) with the same trainer, recipe, data order and global batch. The data order of the resume was verified against the original run (batch hash of zero-based iterations 6751 and 7149, i.e. native steps 6752 and 7150, identical). DDP; global batch 256 × 4096 tokens as in 1.00–1.02 (effective: 4 ranks × micro-batch 2 × gradient accumulation 32) |
| Checkpoints | every 150 steps from 5850 and the final step 7150; all are constant-lr states |
| Trainer | train_gpt_ref_r24_1.py: the 1.02 trainer, releasing the checkpoint tensors from GPU memory after resuming (needed on 24 GB cards); the model definition is unchanged |
| Training curve | steps up to 7150 (6860–7150 from the L40S resume): https://track.fabryka.ai/run/7ca4f84f-3b07-4525-8cde-f3a936dc3425 |
| Benchmarks 1.00–1.03 (lm-eval 0.4.13, 0-shot) | https://track.fabryka.ai/run/1426c0a8-2f07-5a03-80fc-92f7e922aa4b |
Training continues from ckpt_7150.pt on the rest of the prepared corpus (track:
https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5); those checkpoints belong to the next versions.
Benchmark caveat
A leak scan of the whole training superset (SlayerLab/slayer-pl-8x3b@d50df04a, 22 603 122 documents) found n-gram
overlap with BoolQ, MMLU and ARC test items in the English shared part (shared/en: 1428 / 1429 / 1345 documents).
Overlap is not a measured contamination rate; the assessment is in progress. Read the BoolQ, MMLU and ARC numbers with
this caveat. From pack-04 on (first filtered update: zero-based 7192, native step 7193), the 5133 matching documents
are excluded from new training data; earlier steps, including 1.00–1.03, used the unfiltered data.
File format
torch.save dictionary of the trainer: model, opt (Muon + AdamW), step, config. Load with
torch.load(path, map_location="cpu", weights_only=True). Model class: train_gpt_ref_r23.py in the 1.00–1.02 model
repositories.
SHA-256
Recorded off the training process at the moment each checkpoint is written, before upload; files are downloaded back and checked against these sums.
17fd4f09df786f1764a1033dc9dbba50e970197c2a87cffe3434f32729d8776a ckpt_5850.pt
9a801f2363abe7ce68064d891f99d6c3ec7c7098943412e736db668c89790d12 ckpt_6000.pt
4b082cf3b67b95eac50f2379e35b9b0c7ed75bf75260345f77bb4a82803bf580 ckpt_6150.pt
833e1c9baec710e8aab43be0f8623467d66d1498da25764c9fd5f94b118c194c ckpt_6300.pt
065458042ee08ea4c3154256d4405fce33aa4511a48265a11eed6494ed2c849b ckpt_6450.pt
0c290af8dc9892d48d795219b120ab83c4e4e5294338494ab668dc11125410b3 ckpt_6600.pt
970518a13efa13f65bfb6bc17db16b6b9daa010d7bfc615a799d07284d6fa38e ckpt_6750.pt
d094bef832864563e52371c681316220d41ce7415e9c0912cecbfbe81adcd0fe ckpt_6900.pt
f1fbdcfdf124ce961bed5c1ba3949fc61b52bbe1cfee9918655a313482a1b561 ckpt_7050.pt
d7cad9673f3ee706f3022ad6a97fa7593604806d4d25ba936c27885a8f115d97 ckpt_7150.pt
License
Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (shared/core_sa, shared/legal_sa of
slayer-pl-8x3b); see the dataset card for the licenses of the data.