Upload README.md with huggingface_hub

README.md

@@ -98,9 +98,162 @@ Several defects silently break training and are worth documenting:
 
 ---
 
-## 3. Evaluation
+## 3. How to Run Inference
 
-### 3.1 Language modeling perplexity (validation set, identical windows)
+### 3.1 Important: this is a custom architecture
+
+`BaiHu-V1-Flash` uses `model_type: baihu_ssa`, which is **not** in the Transformers
+registry. Loading it with a plain `AutoModelForCausalLM.from_pretrained(...)` fails with:
+
+```
+ValueError: The checkpoint you are trying to load has model type `baihu_ssa`
+but Transformers does not recognize this architecture.
+```
+
+You must register the config and model classes first. This is a one-time, three-line step
+(see below).
+
+Also note: **this model does not go through `transformers.GenerationMixin`.** SSA owns its
+own KV-cache layout (`BaiHuSSACache`) because the cache stores per-block compressed prefix
+sums rather than a plain growing key/value tensor. Call the model's own `generate()` method;
+beam search is not supported.
+
+### 3.2 Setup
+
+```bash
+# The SSA implementation lives in the project repository (not on the Hub),
+# because it is a custom architecture.
+git clone <this-project-repo> ssa_model
+cd ssa_model
+uv venv --python 3.11 .venv
+
+# Ampere or newer (RTX 30xx/40xx, A100, ...): any recent torch works.
+uv pip install --python .venv/Scripts/python.exe \
+    "numpy>=1.26" "transformers>=4.51" "safetensors>=0.4" "torch>=2.6"
+
+# Maxwell / Pascal / Volta (GTX 9xx/10xx, TITAN X, V100, ...): pin torch 2.7.1 —
+# see the GPU note below for why `torch>=2.6` would resolve to a build without your kernels.
+uv pip install --python .venv/Scripts/python.exe \
+    "numpy>=1.26" "transformers>=4.51" "safetensors>=0.4" "torch==2.7.1"
+```
+
+> **GPU note (Maxwell and older).** If you are on an NVIDIA Maxwell card such as the
+> GTX TITAN X (compute capability sm_52), recent PyTorch wheels no longer ship kernels for
+> it: PyTorch **2.8 removed sm_50/sm_60**, and 2.8.0+cu126 only contains `sm_61…sm_90`.
+> You will get `CUDA error: no kernel image is available for execution on the device`.
+> Use **torch 2.7.1+cu126**, which still contains `sm_50`. Also train/infer in **fp32** on
+> Maxwell: measured fp32 5.30 / fp16 4.46 / bf16 3.20 TFLOPS, so half precision is a loss.
+
+### 3.3 Minimal working example
+
+```python
+import os
+import sys
+
+import torch
+from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
+
+# Make the custom implementation importable (path to the cloned repo's src/)
+sys.path.insert(0, os.path.join("ssa_model", "src"))
+
+from baihu_ssa.configuration_baihu_ssa import BaiHuSSAConfig
+from baihu_ssa.model_baihu_ssa import BaiHuSSAForCausalLM
+
+# ---- register the custom architecture (REQUIRED) ----
+AutoConfig.register("baihu_ssa", BaiHuSSAConfig, exist_ok=True)
+AutoModelForCausalLM.register(BaiHuSSAConfig, BaiHuSSAForCausalLM, exist_ok=True)
+
+REPO = "NovaAI6868/BaiHu-V1-Flash"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.float32 if device == "cpu" else torch.float32  # see GPU note: fp32
+
+tokenizer = AutoTokenizer.from_pretrained(REPO)
+model = AutoModelForCausalLM.from_pretrained(REPO, dtype=dtype).to(device).eval()
+
+prompt = "人工智能的未来是"
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
+
+with torch.no_grad():
+    out = model.generate(
+        input_ids,
+        max_new_tokens=32,
+        do_sample=False,        # greedy; supported
+        # do_sample=True, temperature=0.8, top_k=50,   # sampling also supported
+    )
+print(tokenizer.decode(out[0], skip_special_tokens=True))
+```
+
+### 3.4 `generate()` parameters
+
+The built-in `generate()` is a self-contained decoding loop (greedy or
+temperature/top-k sampling). Supported arguments:
+
+| Argument | Default | Meaning |
+|---|---|---|
+| `max_new_tokens` | 32 | number of tokens to generate |
+| `do_sample` | `False` | `False` = greedy, `True` = sample |
+| `temperature` | 1.0 | sampling temperature (used when `do_sample=True`) |
+| `top_k` | `None` | top-k sampling cutoff (used when `do_sample=True`) |
+| `eos_token_id` | `None` | stop early when all sequences emit this token |
+| `use_cache` | `True` | keep the SSA cache between steps; **leave on**, decoding is much slower without it |
+
+**Not supported:** beam search (the SSA cache has no `reorder_cache`), and
+`transformers` generation utilities such as `logits_processor` / `stopping_criteria`.
+
+### 3.5 Computing perplexity / loss
+
+```python
+with torch.no_grad():
+    inputs = tokenizer("Your text here", return_tensors="pt").to(device)
+    loss = model(**inputs, labels=inputs["input_ids"]).loss
+    print("loss", loss.item(), "ppl", torch.exp(loss).item())
+```
+
+### 3.6 Throughput you should expect
+
+Measured on a GTX TITAN X (sm_52, 12.9 GB), fp32, prompt 1024 tokens, 64 generated tokens:
+
+| | Qwen3-0.6B-Base | BaiHu-V1-Flash |
+|---|---|---|
+| Prefill / TTFT | 503 ms | 1431 ms |
+| Per-token decode (TPOT) | 36.6 ms | 80.5 ms |
+| Decode throughput | 27.3 tok/s | 12.4 tok/s |
+| Peak memory (generation) | 9.91 GB | 9.01 GB |
+
+**BaiHu-V1-Flash is currently ~2.2× slower to decode than the base model**, even though the
+sparse path reads fewer keys. The reason is implementation, not architecture: the attention
+layer loops over query blocks in Python and launches many small kernels, so kernel-launch
+overhead dominates the FLOPs saved. Memory use is lower; speed is not yet a win.
+See "Known Limitations" below.
+
+### 3.7 Expected output quality (honest warning)
+
+This checkpoint has seen only **5.0M tokens**, so the new branches are far from converged.
+Greedy decoding tends to **repeat itself** and it is **worse than the base model on
+perplexity** (ppl 82.3 vs 39.4 at 512 tokens). A real greedy sample:
+
+```
+prompt: 人工智能的未来是
+output: 人工智能的未来是怎样的？人工智能的未来是怎样的？人工智能的未来是怎样的？…
+```
+
+Use it to study or continue the SSA retrofit — **do not** expect it to match
+`Qwen3-0.6B-Base` as a general-purpose model yet.
+
+### 3.8 Troubleshooting
+
+| Error | Cause | Fix |
+|---|---|---|
+| `does not recognize this architecture` / `KeyError: 'baihu_ssa'` | custom architecture not registered | call `AutoConfig.register` + `AutoModelForCausalLM.register` as in 3.3 |
+| `CUDA error: no kernel image is available for execution on the device` | installed PyTorch has no kernels for your GPU (Maxwell/sm_52) | install torch 2.7.1+cu126 or older (2.8 dropped sm_50/sm_60) |
+| `AttributeError: ... has no attribute 'tie_weights'` | an external tool treating the model as `PreTrainedModel` | use a version of the implementation that defines `tie_weights()` (already fixed upstream) |
+| `BaiHuSSACache` errors when calling `model.generate(...)` from GenerationMixin | this model bypasses `GenerationMixin` | call `model.generate(...)` on the BaiHu model itself |
+
+---
+
+## 4. Evaluation
+
+### 4.1 Language modeling perplexity (validation set, identical windows)
 
 | Sequence length | Qwen3-0.6B-Base | BaiHu-V1-Flash |
 |---|---|---|
@@ -108,7 +261,7 @@ Several defects silently break training and are worth documenting:
 | 1024 | 3.4691 / ppl 32.109 | 4.2575 / ppl 70.631 |
 | 2048 | 3.0801 / ppl 21.761 | 3.9213 / ppl 50.467 |
 
-### 3.2 Standard benchmarks (lm-evaluation-harness)
+### 4.2 Standard benchmarks (lm-evaluation-harness)
 
 | Task | Qwen3-0.6B-Base | BaiHu-V1-Flash | Delta |
 |---|---|---|---|
@@ -117,7 +270,7 @@ Several defects silently break training and are worth documenting:
 | piqa | 0.7050 | 0.6950 | -0.0100 |
 | winogrande | 0.6300 | 0.6300 | +0.0000 |
 
-### 3.3 Inference compute and resource usage
+### 4.3 Inference compute and resource usage
 
 | Metric | Qwen3-0.6B-Base | BaiHu-V1-Flash |
 |---|---|---|
@@ -157,7 +310,7 @@ Negative findings, stated plainly:
   (+0.070 acc_norm), winogrande is unchanged, while hellaswag (−0.015) and piqa (−0.010)
   regress slightly.
 
-### 3.4 Sparsity
+### 4.4 Sparsity
 
 Two measurements are reported because they answer different questions and are **not**
 interchangeable:
@@ -170,15 +323,15 @@ interchangeable:
 The first number averages over *all* query blocks including the early ones, which have no
 completed blocks available to select and therefore read nothing through the sparse path.
 The second is the steady-state ratio for later queries, and it is the one that matters for
-efficiency — see the note in section 3.3: at these sequence lengths the sparse path is not
+efficiency — see the note in section 4.3: at these sequence lengths the sparse path is not
 yet saving meaningful work.
 
 ---
 
-## 4. Known Limitations
+## 5. Known Limitations
 
 1. **Trained for very little.** Only 5.0M tokens (≈0.25 epoch). The new branches have not
-   converged; ppl is well above the base model and decode is slower (see 3.3). Continuing
+   converged; ppl is well above the base model and decode is slower (see 3.6 and 4.1). Continuing
    to 200M tokens or more is required before the SSA layers can genuinely take over
    long-range modeling.
 2. **The shared branch does not appear to help and was actively suppressed by the
@@ -206,7 +359,7 @@ yet saving meaningful work.
 
 ---
 
-## 5. License
+## 6. License
 
 **Free for personal use; a paid license is required for commercial use.**
 
@@ -225,7 +378,7 @@ component's original license.
 
 ---
 
-## 6. Citation
+## 7. Citation
 
 ```bibtex
 @misc{baihu-v1-flash,