ForgePlex-M2.1-10M

ForgePlex-M2.1-10M is a 9.99M-parameter decoder-only language model from ForgeWorks, and the most complete model we've built so far. We thank the ForgeWorks team alongside anyone who delivered assistance in the creation of this model.

Why not name it ForgePlex-M3?

We decided to give the model the M2.1 classification because at its core it's still M2. The changes are refinements: a slightly wider feed-forward layer (707 to 712), a retrained tokenizer, and a longer, better-tuned training run. A new major version should mean a real architectural step, so we're saving the M3 name for that.

Metric	Value
Unique parameters	9,991,938
Intelligence Index	10.70
HellaSwag	29.09%
ARC (combined)	29.66%
PIQA	59.14%
ArithMark-3	35.50%
License	Apache-2.0

Architecture

GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style attention output gates on every layer and GPT-S2-style refresh gates on inject layers [5, 10] (kernel 9). XSA is off. Weights keep the original key layout (no Llama remapping).

Component	Details
Position encoding	RoPE (theta=5,000, NeoX even/odd)
Normalization	RMSNorm (eps=1e-6)
Feed-forward	SwiGLU (intermediate 712)
Attention	GQA: 8Q / 2KV, head_dim=32 + attn output gate
Refresh	Layers 5, 10, kernel 9
Bias	None
Embedding	Weight tying
Depth x width	11 layers x 256 hidden
Context	1024 tokens
Vocab	4,096 custom BPE

ForgePlex-M2.1-10M vs <10m Leaders

ForgePlex-M2.1 leads the current <10M top models on HellaSwag and PIQA, and stays competitive on ARC and ArithMark. M2.1 improves on M2 in HellaSwag, PIQA and ArithMark, and holds ARC roughly level. ## Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ForgeWorks/ForgePlex-M2.1-10M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Or run python usage.py from a local copy of this repo.

@misc{forgeplex_m21_10m,
  title={ForgePlex-M2.1-10M},
  author={ForgeWorks},
  year={2026},
  howpublished={\url{https://huggingface.co/ForgeWorks/ForgePlex-M2.1-10M}}
}