PASITA v1 — plain text to Markdown (faithful)

A decoder-only language model trained 100% from scratch (no base model), specialized in a single task: converting plain text into valid Markdown while preserving information.

Compiler behavior, not chatbot behavior: it adds structure, it does not invent content.

Architecture (lab notes)

Parameter	Value
Class	LlamaForCausalLM (decoder-only, dense, no MoE)
Parameters	88,099,584 (~88M) in bfloat16
Layers / hidden / FFN	12 / 768 / 2048 (SwiGLU)
Attention	GQA 12Q/4KV, head_dim 64, no bias
Positions	RoPE theta=100000, ctx 2048, RMSNorm eps=1e-5
Embeddings	tied (saves ~12.6M params)
File	model.safetensors (176 MB, 110 tensors, sha 664665e7…)
Tokenizer	Custom 16k byte-level BPE, verified ByteLevel decoder (~4.0 chars/token ES/EN)
Special tokens	<pad> <s> </s> <unk> <think> </think>

Training

1. SFT 58M tokens x2 epochs — final loss 0.09, token accuracy 98.4%
2. DPO beta=0.1 — preference margin 4.2
3. GRPO 550+150 steps, G=4, verifiable rewards (format + numeric fidelity + anti-overformatting)
4. Data: 80M human markdown-derived tokens (ES/EN Wikipedia, StackExchange, WikiHow) scaled to 625M in v4 corpus

Benchmark (held-out n=1000, greedy)

Global	GFM 0.956 - faith 0.927 - sem 0.888 - tables 1.0
code / ocr / docs / math / tables / html	0.93 - 1.00
control (strict instructions)	0.19 (known limitation)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("OpceanAI/PASITA")
model = AutoModelForCausalLM.from_pretrained("OpceanAI/PASITA", dtype="auto")
prompt = "CONVIERTE A MARKDOWN:\n" + text + "\n\n### Markdown:\n"
ids = tok(prompt, return_tensors="pt", truncation=True, max_length=1024)
out = model.generate(**ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

Valid regime: medium/long documents (OCR output, pasted HTML, meeting notes, tutorials). Fragile on 1-3 line inputs.

Limitations

May truncate digits, drop secondary data, emit echo H1s, or continue past completion. Recommended: adaptive max_new_tokens + beam search + fidelity rerank.