Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,314 @@
+---
+license: apache-2.0
+license_link: https://ai.google.dev/gemma/docs/gemma_4_license
+base_model: google/gemma-4-26B-A4B-it
+base_model_relation: adapter
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-classification
+tags:
+  - system-one
+  - system-two
+  - adaptive-thinking
+  - typed-decisions
+  - decision-model
+  - calibrated-probabilities
+  - jev
+  - noul
+  - choice
+  - score
+  - lora
+  - gemma4
+  - mixture-of-experts
+  - multimodal
+  - vllm
+model-index:
+  - name: autotrust/GEV-26B-Decide
+    results:
+      - task:
+          type: text-classification
+          name: Jev Decision Index 0.2.1, System 1 (complete run, 150,317 scored requests)
+        dataset:
+          type: decision-index
+          name: Decision Index suite 0.2 (edition 0.2.1)
+        metrics:
+          - type: decision_index
+            name: Decision Index (balanced skill)
+            value: 58.05
+---
+
+# autotrust/GEV-26B-Decide
+
+### A decision model with adaptive thinking, on Gemma-4-26B-A4B-it (26 B parameters, ≈ 4 B active per token)
+
+**GEV-26B-Decide answers typed questions with a calibrated probability for every option, and thinks only when it needs
+to.** System 1 decides in one forward pass (about 45 ms). When its leading option is uncertain, System 2 (the same
+backbone in Gemma-4 thinking mode) reasons over the question, and the reasoning is folded into the final probabilities.
+One set of weights, one vLLM engine, for text and images.
+
+| | what it does | output |
+|---|---|---|
+| **System 1** | typed decisions: yes/no · pick one of 2–256 options · rate 0–5, over text and images; prompts up to 256K tokens | a calibrated probability for every option, in one forward pass |
+| **Adaptive thinking** | System 1 first; below 0.8 confidence, System 2 thinks and its answer is folded in | calibrated probabilities |
+| **System 2** | the unmodified `google/gemma-4-26B-A4B-it`, optionally thinking step by step, text and images | text / reasoning |
+
+GEV-26B-Decide was previously published as `autotrust/JEV-Gemma4-26B-A4B`; the weights are the same.
+
+> **Two models, two organisations.** **TypeSafe Jev 1.13** is the hosted, closed model made by TypeSafe AI.
+> **autotrust/GEV-26B-Decide** is an independent open-weights model built by AutoTrust AI; it is not affiliated with,
+> endorsed by, or a product of TypeSafe AI.
+
+## Adaptive thinking
+
+1. **Fast distribution.** System 1 returns p1 in one pass.
+2. **Think only when uncertain.** If the leading option of p1 is below the threshold (default 0.8), System 2 reasons in
+   Gemma-4's thinking mode over the same state, question and options (budget 8,192 thinking tokens).
+3. **Fold the reasoning in.** When the thinking channel closes, the answer-letter distribution p2 is read in one step,
+   and the result is p = ½ p1 + ½ p2. On its own, p2 is close to one-hot and over-confident; the equal mix keeps the
+   reasoning's accuracy and System 1's calibration.
+
+The threshold and the mix were chosen on 1,754 questions from six public sets that are not part of the Decision Index
+(test or validation splits, 300 random questions each; AQuA-RAT has 254):
+
+| set | System 1 | **adaptive** | thinking on | always think |
+|---|---:|---:|---:|---:|
+| AQuA-RAT (math word problems) | 68.1 | **89.0** | 53.9 % | 90.2 |
+| LogiQA (logical reasoning) | 55.3 | **80.3** | 63.7 % | 82.0 |
+| StrategyQA (multi-hop yes/no) | 68.0 | **78.7** | 71.0 % | 79.0 |
+| MedMCQA (medical) | 66.7 | **71.7** | 52.3 % | 73.7 |
+| OpenBookQA (science) | 94.3 | **96.3** | 14.7 % | 95.7 |
+| CommonsenseQA | 86.7 | **85.0** | 32.0 % | 83.3 |
+| **all 1,754** | 73.3 | **83.4** | 47.8 % | 83.8 |
+
+Accuracy in %. Calibration is unchanged: ECE 0.035 for System 1 and 0.035 for adaptive. The adaptive mode reaches 96 % of
+the always-think gain while thinking on 48 % of the questions. Thinking length: median 2,282 tokens, 90th percentile 7,573;
+91.6 % of the thoughts finish within the 8,192-token budget.
+
+**Decision Index, Knowledge & Reasoning with adaptive thinking:** the full run of the area's ten benchmarks (33,846
+requests) is in progress; its results will be added here.
+
+## Decision Index 0.2.1 (System 1)
+
+Complete run of all 150,759 requests of suite 0.2 (150,317 scored; 0 errors, 0 unsupported) with System 1 only, scored
+with the kit's `score --edition 0.2.1`. Results:
+[`autotrust/jev-decision-index-results`](https://huggingface.co/datasets/autotrust/jev-decision-index-results)
+(`runs/jev-gemma4-26b-a4b`, the run of these weights under their previous name).
+
+| | Decision Index (balanced skill) | balanced raw | breadth skill |
+|---|---:|---:|---:|
+| **autotrust/GEV-26B-Decide** (System 1) | **58.05** | 67.35 | 56.98 |
+| TypeSafe Jev 1.13 (board) | 57.91 | — | — |
+| [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B) | 53.30 | 64.32 | 52.21 |
+
+| area (skill) | Knowledge & Reasoning | Language | Retrieval & Classification | Tools & Automation | Arts & Taste |
+|---|---:|---:|---:|---:|---:|
+| GEV-26B-Decide (System 1) | 0.430 | 0.636 | 0.679 | 0.697 | 0.415 |
+
+## Images
+
+The checkpoint contains Gemma-4's vision encoder (no audio encoder), so System 1 and System 2 both accept images. The
+decision head was trained on text; decisions over images are zero-shot.
+
+| check | result |
+|---|---|
+| synthetic images: colour (8 options), shape (4), printed number (8), "is there a red object?" (yes/no) | 100 % on each (30 images each) |
+| [VL-RewardBench](https://huggingface.co/datasets/MMInstruction/VL-RewardBench), 1,247 pairs, both presentation orders averaged | **78.4 %** overall (general 55.8, hallucination 84.9, reasoning 76.0; macro 72.2) |
+
+For reference, [autotrust/JEV-27B-VL](https://huggingface.co/autotrust/JEV-27B-VL) scores 78.3 % on VL-RewardBench with
+the same protocol.
+
+## Context length
+
+The backbone's native context is **262,144 tokens (256K)**. We tested decisions that hinge on a single sentence placed at
+a random depth in long real text (concatenated PubMedQA abstracts): a yes/no question and a 16-option question, 10 of
+each per length, on one B200 with vLLM, one request at a time.
+
+| prompt length | yes/no correct | 16-option correct | mean probability on the right answer | median latency |
+|---|---:|---:|---:|---:|
+| 4K | 10/10 | 10/10 | 0.999 | 0.15 s |
+| 32K | 10/10 | 10/10 | 0.999 | 1.6 s |
+| 64K | 10/10 | 10/10 | 0.999 | 5.0 s |
+| 128K | 10/10 | 10/10 | 1.000 | 17.8 s |
+
+Lengths above 128K have not been tested yet.
+
+## Many options
+
+`choice` takes 2–256 options. Up to 16 are read in one pass with the trained labels A–P. More options are read in groups
+of at most 16 (in parallel), then a final of 16; every option is read and none is pruned (`strategy: "tournament"`, the
+default). Zero-shot intent classification, all options offered at once, 400 test utterances per row:
+
+| test | options | accuracy |
+|---|---:|---:|
+| MASSIVE (en) | 59 | 91.2 % |
+| BANKING77 | 77 | 81.5 % |
+| CLINC150 | 150 | 95.5 % |
+| CLINC150 utterances among CLINC150 + BANKING77 + MASSIVE intents | 255 | 89.0 % |
+| BANKING77 utterances among the same 255 intents | 255 | 75.2 % |
+
+The 255-option sets merge three catalogues with overlapping intents, so part of the drop comes from near-duplicate labels.
+A single pass with labels beyond P (`strategy: "single"`) is about 3× faster but less accurate here (CLINC150: 88.5 %
+against 95.2 %), so it is not the default. The BANKING77 and CLINC150 training splits are part of the training data (see
+below).
+
+## Quick start (vLLM)
+
+```bash
+hf download autotrust/GEV-26B-Decide --local-dir GEV-26B-Decide
+bash GEV-26B-Decide/serve.sh          # vLLM on :8000; one GPU with 80 GB or more
+```
+
+`serve.sh` runs `serve_decide.py`: the standard vLLM OpenAI server (same flags as `vllm serve`) with a `POST /v1/decide`
+route. It loads the backbone once: plain requests are System 2, and requests for the LoRA module `jev-decision`
+(`adapter_vllm/`: backbone LoRA + the decision head as an `lm_head` LoRA) are System 1. It needs a vLLM build with
+Gemma-4 support plus `patches/vllm-gemma4-lm-head-lora.patch` (LoRA on Gemma-4's tied `lm_head`, vocabulary 262,144);
+tested with a vLLM development build from September 2026.
+
+### System 1 and adaptive thinking: `POST /v1/decide`
+
+```bash
+curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{
+  "kind": "choice",
+  "state": "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.",
+  "question": "How much does the ball cost?",
+  "options": ["$0.10", "$0.05", "$1.00", "$0.55"],
+  "thinking": "auto"}'
+```
+
+| field | value |
+|---|---|
+| `kind` | `noul`: yes/no, probabilities for `["false", "true"]` · `score`: 0–5 · `choice`: your `options` |
+| `state` | what the decision is about: a string, a JSON object, or a list mixing text and images `["Photo: ", {"image": "https://… or data:…"}]` |
+| `question` | one question about the state |
+| `options` | `choice` only: 2–256 strings |
+| `thinking` | `"off"` (default: System 1 only), `"auto"` (adaptive), `"on"` (always think); `noul` and `choice` |
+| `threshold` | System 1 confidence below which `"auto"` thinks (default 0.8) |
+| `think_budget` | maximum thinking tokens (default 8,192) |
+| `strategy` | more than 16 options: `"tournament"` (default), `"single"`, `"permute"` |
+| `return_reasoning` / `debug` | include System 2's reasoning / the System 1 and System 2 distributions |
+
+The response has `options`, `probabilities`, `choice`, `choice_index`, `usage` and, when thinking was requested,
+`thinking: {"used": true, "think_tokens": …, "think_seconds": …}`. `GET /v1/decide/info` lists the defaults.
+
+```python
+import requests
+
+def decide(kind, state, question, options=None, thinking="auto"):
+    body = {"kind": kind, "state": state, "question": question, "thinking": thinking, **({"options": options} if options else {})}
+    r = requests.post("http://localhost:8000/v1/decide", json=body).json()
+    return dict(zip(r["options"], r["probabilities"])), r.get("thinking", {}).get("used")
+
+decide("noul", "John was born on 29 February 1996.", "Was John's 7th birthday celebrated on a 29 February?")
+```
+
+### System 2
+
+```python
+requests.post("http://localhost:8000/v1/chat/completions", json={
+    "model": "autotrust/GEV-26B-Decide",
+    "messages": [{"role": "user", "content": "In one sentence, what is safety stock?"}],
+    "max_tokens": 200, "chat_template_kwargs": {"enable_thinking": False}})
+```
+
+### Speed (one B200, vLLM)
+
+* System 1: median 45 ms for a single request; 257 decisions per second with 64 concurrent clients. vLLM matches the
+  `transformers` engine below to a mean largest probability difference of 0.015 (300 held-out decisions).
+* Adaptive thinking adds a few seconds when it thinks (median 2,282 thinking tokens) and nothing when it does not.
+
+If you call `/v1/completions` for System 1 yourself, pass `top_k: 0` and `top_p: 1.0`: the model's generation config sets
+`top_k=64` and `top_p=0.95`, which vLLM applies as request defaults and which would truncate the returned probabilities.
+
+## Usage (transformers + peft, System 1)
+
+```python
+import json, torch
+from huggingface_hub import snapshot_download
+from peft import PeftModel
+from safetensors.torch import load_file
+from transformers import AutoTokenizer, Gemma4ForConditionalGeneration
+
+d = snapshot_download("autotrust/GEV-26B-Decide")
+tok = AutoTokenizer.from_pretrained(d)
+base = Gemma4ForConditionalGeneration.from_pretrained(d, dtype=torch.bfloat16, device_map="cuda")
+# System 2: `base` is gemma-4-26B-A4B-it unchanged; use base.generate(...) (text or images).
+
+# System 1: adapter merged in memory + head
+m = PeftModel.from_pretrained(base, f"{d}/adapter").merge_and_unload().eval()
+backbone = m.model
+jc, T = json.load(open(f"{d}/judge_config.json")), json.load(open(f"{d}/calibration.json"))["per_kind"]
+head = load_file(f"{d}/head.safetensors"); W, b = head["proj.weight"].cuda(), head["proj.bias"].cuda()
+
+@torch.no_grad()
+def decide(kind, state, question, options):
+    lines = options if kind != "choice" else [f"{'ABCDEFGHIJKLMNOP'[i]}) {o}" for i, o in enumerate(options)]
+    text = f"[kind] {kind}\n[state] {state}\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"
+    ids = torch.tensor([[tok.bos_token_id] + tok.encode(text, add_special_tokens=False)], device="cuda")
+    h = backbone(input_ids=ids, use_cache=False).last_hidden_state[0, -1].float()
+    z = 30.0 * torch.tanh((W @ h + b) / 30.0)
+    s, _ = jc["slots"]["ranges"][kind]
+    return dict(zip(options, torch.softmax(z[s:s + len(options)] / T[kind], 0).tolist()))
+
+print(decide("noul", "Customer says the parcel arrived damaged and wants their money back.",
+             "Is the customer asking for a refund?", ["false", "true"]))
+```
+
+This path reads up to 16 options per pass; for more, use the server (or read groups of 16 and a final, as above).
+
+## Engine and read-out
+
+* **Template** `bare-v1`, prefixed with `<bos>`: `[kind] … [state] … [question] … [options] A) … [decision]:`
+* **Read-out**: the final-norm hidden state of the last token goes through a linear fp32 head (hidden 2,816 → 24 slots),
+  soft-capped at 30 like Gemma's own logits; inactive slots are masked, the logits are divided by the per-kind
+  temperature and softmaxed. Nothing is generated.
+
+## Temperatures
+
+| table | noul | choice | score |
+|---|---|---|---|
+| `calibration.json` (default; used in the Decision Index run) | 1.003 | 1.017 | 0.999 |
+| `calibration_gold.json` (calibrated against ground-truth answers) | 1.214 | 1.098 | 1.000 |
+
+Use `calibration_gold.json` when you gate automatic actions on confidence.
+
+## Training data (disclosure)
+
+System 1 was trained on teacher distributions and ground-truth decision data. The ground-truth data includes the public
+**training splits** of some datasets whose **test** splits the Decision Index uses (among them BANKING77 and CLINC150);
+no test split of any benchmark was used, and suite items were excluded before training. The list has been provided to
+the Decision Index maintainers. MMMU / MMMU-Pro are not valid evaluations for this model. The adaptive-thinking settings
+were chosen on data outside the Decision Index suite.
+
+## Limitations
+
+* Where the teacher is wrong, System 1 often is too; adaptive thinking helps most on math, logic and multi-step
+  questions, and can slightly lower accuracy on commonsense questions (CommonsenseQA 86.7 → 85.0).
+* Thinking is slow on hard inputs: on expert-level and chess questions it often uses the full 8,192-token budget.
+* Weaker than JEV-27B on long structured inputs (e.g. the Decision Index's Home appliance simulator and POP909).
+* HLE: below chance with System 1, like every open entry on the board.
+* Decisions over images are zero-shot; contexts above 128K tokens are untested.
+* The one-engine vLLM setup needs the bundled vLLM patch; `noul` and `score` accept only their canonical options.
+* English-centric; not for high-stakes decisions without confidence gating.
+
+## Files
+
+```
+model-*.safetensors · config.json · processor_config.json · tokenizer* · chat_template.jinja · generation_config.json
+                          google/gemma-4-26B-A4B-it, unchanged (System 2; text + image input)
+adapter/                  System 1 LoRA (peft), for the transformers path
+head.safetensors          24-slot decision head (fp32): proj.weight [24, 2816], proj.bias [24]
+judge_config.json         slot layout, verbalizer ids, softcap, read-out
+calibration.json          per-kind temperatures (default)
+calibration_gold.json     per-kind temperatures calibrated against ground-truth answers
+adapter_vllm/             System 1 for vLLM: backbone LoRA + the head as an lm_head LoRA, plus decision_head.json
+serve_decide.py · serve.sh
+                          vLLM server with POST /v1/decide (System 1, adaptive thinking) next to the OpenAI endpoints
+patches/                  vLLM patch: LoRA on Gemma-4's tied lm_head
+reports/                  Decision Index scores (System 1); adaptive-thinking validation summary
+```
+
+## License
+
+Apache-2.0 for the adapter, head and calibration files; base model under the Gemma 4 terms
+(<https://ai.google.dev/gemma/docs/gemma_4_license>). Not affiliated with TypeSafe AI.

adapter/README.md

@@ -0,0 +1,5 @@
+# System 1 adapter (PEFT LoRA)
+
+LoRA adapter for [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it), part of
+[autotrust/GEV-26B-Decide](https://huggingface.co/autotrust/GEV-26B-Decide). Use it with `head.safetensors`, `judge_config.json`
+and `calibration.json` from the repository root; see the model card. `adapter_vllm/` is the same System 1 for vLLM.

adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma4ForConditionalGeneration",
+    "parent_library": "transformers.models.gemma4.modeling_gemma4"
+  },
+  "base_model_name_or_path": "google/gemma-4-26B-A4B-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "kasa_config": null,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.21.0",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "model\\.language_model\\.layers\\.\\d+\\.(self_attn\\.(q|k|v|o)_proj|mlp\\.(gate|up|down)_proj)",
+  "target_parameters": null,
+  "task_type": null,
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94fb34ad3b58266c304a58d1675ae4a9f02f3be453d08135e185c8079236836e
+size 148745744

adapter_vllm/adapter_config.json

@@ -0,0 +1,55 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma4ForConditionalGeneration",
+    "parent_library": "transformers.models.gemma4.modeling_gemma4"
+  },
+  "base_model_name_or_path": "google/gemma-4-26B-A4B-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "kasa_config": null,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64.0,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.21.0",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "gate_proj",
+    "k_proj",
+    "lm_head",
+    "o_proj",
+    "q_proj",
+    "up_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": null,
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

adapter_vllm/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c20767b1a3fd8717d5cf709a98276666d2c6d7e84e973026a07091343294291
+size 92591488

adapter_vllm/decision_head.json

@@ -0,0 +1,101 @@
+{
+ "bias": [
+  0.004050370771437883,
+  0.0025548420380800962,
+  -0.0009425673633813858,
+  -0.00034990705898962915,
+  0.0007876436575315893,
+  0.004409449640661478,
+  0.002710630651563406,
+  -0.0009728983859531581,
+  -0.0007952140877023339,
+  -0.0021821260452270508,
+  -0.001166780013591051,
+  -0.00140042370185256,
+  -0.0026794816367328167,
+  -0.003741167951375246,
+  -0.003188050352036953,
+  -0.006115781608968973,
+  -0.0021734880283474922,
+  -0.0018658898770809174,
+  -0.0028179585933685303,
+  -0.00424833782017231,
+  -0.004550046753138304,
+  -0.004217465873807669,
+  -0.005448763258755207,
+  -0.005735703278332949
+ ],
+ "verbalizer_ids": [
+  4530,
+  3397,
+  236771,
+  236770,
+  236778,
+  236800,
+  236812,
+  236810,
+  236776,
+  236799,
+  236780,
+  236796,
+  236788,
+  236811,
+  236823,
+  236814,
+  236777,
+  236863,
+  236855,
+  236798,
+  236792,
+  236797,
+  236806,
+  236791
+ ],
+ "slots": {
+  "num_slots": 24,
+  "ranges": {
+   "noul": [
+    0,
+    2
+   ],
+   "score": [
+    2,
+    8
+   ],
+   "choice": [
+    8,
+    24
+   ]
+  },
+  "verbalizers": [
+   "false",
+   "true",
+   "0",
+   "1",
+   "2",
+   "3",
+   "4",
+   "5",
+   "A",
+   "B",
+   "C",
+   "D",
+   "E",
+   "F",
+   "G",
+   "H",
+   "I",
+   "J",
+   "K",
+   "L",
+   "M",
+   "N",
+   "O",
+   "P"
+  ],
+  "template_version": "bare-v1"
+ },
+ "softcap": 30.0,
+ "prefix_token": 2,
+ "note": "decision logits = lm_head-LoRA logprobs of verbalizer ids + bias, then per-kind temperature (Gemma: vLLM applies the model's final-logit softcap; |bias| is tiny, so adding it after the cap is exact to ~1e-2)"
+}

calibration.json

@@ -0,0 +1,44 @@
+{
+  "version": 1,
+  "per_kind": {
+    "noul": 1.0027465035537586,
+    "choice": 1.017196267473986,
+    "score": 0.9992906948414192
+  },
+  "per_kind_family": {},
+  "fit": {
+    "noul": {
+      "n": 4363,
+      "T": 1.0027465035537586,
+      "kl_before": 0.007547731976956129,
+      "kl_after": 0.007547072134912014
+    },
+    "choice": {
+      "n": 3418,
+      "T": 1.017196267473986,
+      "kl_before": 0.06508523225784302,
+      "kl_after": 0.06501992046833038
+    },
+    "score": {
+      "n": 3173,
+      "T": 0.9992906948414192,
+      "kl_before": 0.035448525100946426,
+      "kl_after": 0.035448361188173294
+    }
+  },
+  "diagnostic_calibration_split": {
+    "raw": {
+      "kl": 0.03358318656682968,
+      "ece": 0.001969979859321292,
+      "mce": 0.01303844153881073
+    },
+    "calibrated": {
+      "kl": 0.033562496304512024,
+      "ece": 0.0012317931094873945,
+      "mce": 0.012980207800865173
+    }
+  },
+  "source": "checkpoints/s2_gemma4_step7000",
+  "fit_rows": 10954,
+  "d1_excluded": true
+}

calibration_gold.json

@@ -0,0 +1,38 @@
+{
+  "version": 1,
+  "per_kind": {
+    "noul": 1.2136210312191784,
+    "choice": 1.0979579333985496,
+    "score": 1.0
+  },
+  "per_kind_family": {},
+  "fit": {
+    "noul": {
+      "n": 763,
+      "T": 1.2136210312191784,
+      "kl_before": 0.3925037682056427,
+      "kl_after": 0.38618776202201843
+    },
+    "choice": {
+      "n": 17929,
+      "T": 1.0979579333985496,
+      "kl_before": 0.318155437707901,
+      "kl_after": 0.31654489040374756
+    }
+  },
+  "diagnostic_calibration_split": {
+    "raw": {
+      "kl": 0.3211902976036072,
+      "ece": 0.013034004604066436,
+      "mce": 0.0645633339881897
+    },
+    "calibrated": {
+      "kl": 0.31938767433166504,
+      "ece": 0.005258639238451115,
+      "mce": 0.8051199913024902
+    }
+  },
+  "source": "checkpoints/s2_gemma4_step7000",
+  "fit_rows": 18692,
+  "d1_excluded": true
+}

chat_template.jinja

@@ -0,0 +1,390 @@
+{#
+  Template: Google Gemma 4 Canonical Chat Template
+  Author: Google Gemma Engineering Team
+  Published: 2026-07-09
+  Context: Fixed tool-calling loops, turn closures, and thinking content-ordering.
+#}
+{%- macro format_parameters(properties, required, filter_keys=false) -%}
+    {%- set standard_keys = ['description', 'type', 'properties', 'required', 'nullable'] -%}
+    {%- set ns = namespace(found_first=false) -%}
+    {%- for key, value in properties | dictsort -%}
+        {%- set add_comma = false -%}
+        {%- if not filter_keys or key not in standard_keys -%}
+            {%- if ns.found_first %},{% endif -%}
+            {%- set ns.found_first = true -%}
+            {{ key }}:{
+            {%- if value['description'] -%}
+                description:<|"|>{{ value['description'] }}<|"|>
+                {%- set add_comma = true -%}
+            {%- endif -%}
+            {%- if value['type'] | upper == 'STRING' -%}
+                {%- if value['enum'] -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    enum:{{ format_argument(value['enum']) }}
+                {%- endif -%}
+            {%- elif value['type'] | upper == 'ARRAY' -%}
+                {%- if value['items'] is mapping and value['items'] -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    items:{
+                    {%- set ns_items = namespace(found_first=false) -%}
+                    {%- for item_key, item_value in value['items'] | dictsort -%}
+                        {%- if item_value is not none -%}
+                            {%- if ns_items.found_first %},{% endif -%}
+                            {%- set ns_items.found_first = true -%}
+                            {%- if item_key == 'properties' -%}
+                                properties:{
+                                {%- if item_value is mapping -%}
+                                    {{- format_parameters(item_value, value['items']['required'] | default([])) -}}
+                                {%- endif -%}
+                                }
+                            {%- elif item_key == 'required' -%}
+                                required:[
+                                {%- for req_item in item_value -%}
+                                    <|"|>{{- req_item -}}<|"|>
+                                    {%- if not loop.last %},{% endif -%}
+                                {%- endfor -%}
+                                ]
+                            {%- elif item_key == 'type' -%}
+                                {%- if item_value is string -%}
+                                    type:{{ format_argument(item_value | upper) }}
+                                {%- else -%}
+                                    type:{{ format_argument(item_value | map('upper') | list) }}
+                                {%- endif -%}
+                            {%- else -%}
+                                {{ item_key }}:{{ format_argument(item_value) }}
+                            {%- endif -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                    }
+                {%- endif -%}
+            {%- endif -%}
+            {%- if value['nullable'] %}
+                {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                nullable:true
+            {%- endif -%}
+            {%- if value['type'] | upper == 'OBJECT' -%}
+                {%- if value['properties'] is defined and value['properties'] is mapping -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    properties:{
+                    {{- format_parameters(value['properties'], value['required'] | default([])) -}}
+                    }
+                {%- elif value is mapping -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    properties:{
+                    {{- format_parameters(value, value['required'] | default([]), filter_keys=true) -}}
+                    }
+                {%- endif -%}
+                {%- if value['required'] -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    required:[
+                    {%- for item in value['required'] | default([]) -%}
+                        <|"|>{{- item -}}<|"|>
+                        {%- if not loop.last %},{% endif -%}
+                    {%- endfor -%}
+                    ]
+                {%- endif -%}
+            {%- endif -%}
+            {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+            type:<|"|>{{ value['type'] | upper }}<|"|>}
+        {%- endif -%}
+    {%- endfor -%}
+{%- endmacro -%}
+{%- macro format_function_declaration(tool_data) -%}
+    declaration:{{- tool_data['function']['name'] -}}{description:<|"|>{{- tool_data['function']['description'] -}}<|"|>
+    {%- set params = tool_data['function']['parameters'] -%}
+    {%- if params -%}
+        ,parameters:{
+        {%- if params['properties'] -%}
+            properties:{ {{- format_parameters(params['properties'], params['required']) -}} },
+        {%- endif -%}
+        {%- if params['required'] -%}
+            required:[
+            {%- for item in params['required'] -%}
+                <|"|>{{- item -}}<|"|>
+                {{- ',' if not loop.last -}}
+            {%- endfor -%}
+            ],
+        {%- endif -%}
+        {%- if params['type'] -%}
+            type:<|"|>{{- params['type'] | upper -}}<|"|>}
+        {%- endif -%}
+    {%- endif -%}
+    {%- if 'response' in tool_data['function'] -%}
+        {%- set response_declaration = tool_data['function']['response'] -%}
+        ,response:{
+        {%- if response_declaration['description'] -%}
+            description:<|"|>{{- response_declaration['description'] -}}<|"|>,
+        {%- endif -%}
+        {%- if response_declaration['type'] | upper == 'OBJECT' -%}
+            type:<|"|>{{- response_declaration['type'] | upper -}}<|"|>}
+        {%- endif -%}
+    {%- endif -%}
+    }
+{%- endmacro -%}
+{%- macro format_argument(argument, escape_keys=True) -%}
+    {%- if argument is none -%}
+        {{- 'null' -}}
+    {%- elif argument is string -%}
+        {{- '<|"|>' + argument + '<|"|>' -}}
+    {%- elif argument is boolean -%}
+        {{- 'true' if argument else 'false' -}}
+    {%- elif argument is mapping -%}
+        {{- '{' -}}
+        {%- set ns = namespace(found_first=false) -%}
+        {%- for key, value in argument | dictsort -%}
+            {%- if ns.found_first %},{% endif -%}
+            {%- set ns.found_first = true -%}
+            {%- if escape_keys -%}
+                {{- '<|"|>' + key + '<|"|>' -}}
+            {%- else -%}
+                {{- key -}}
+            {%- endif -%}
+            :{{- format_argument(value, escape_keys=escape_keys) -}}
+        {%- endfor -%}
+        {{- '}' -}}
+    {%- elif argument is sequence -%}
+        {{- '[' -}}
+        {%- for item in argument -%}
+            {{- format_argument(item, escape_keys=escape_keys) -}}
+            {%- if not loop.last %},{% endif -%}
+        {%- endfor -%}
+        {{- ']' -}}
+    {%- else -%}
+        {{- argument -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro strip_thinking(text) -%}
+    {%- set ns = namespace(result='') -%}
+    {%- for part in text.split('<channel|>') -%}
+        {%- if '<|channel>' in part -%}
+            {%- set ns.result = ns.result + part.split('<|channel>')[0] -%}
+        {%- else -%}
+            {%- set ns.result = ns.result + part -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- ns.result | trim -}}
+{%- endmacro -%}
+
+{%- macro format_tool_response_block(tool_name, response) -%}
+    {{- '<|tool_response>' -}}
+    {%- if response is mapping -%}
+        {{- 'response:' + tool_name + '{' -}}
+        {%- for key, value in response | dictsort -%}
+            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}
+            {%- if not loop.last %},{% endif -%}
+        {%- endfor -%}
+        {{- '}' -}}
+    {%- else -%}
+        {{- 'response:' + tool_name + '{value:' + format_argument(response, escape_keys=False) + '}' -}}
+    {%- endif -%}
+    {{- '<tool_response|>' -}}
+{%- endmacro -%}
+
+{#- ===== SETUP ===== -#}
+{%- set ns = namespace(prev_message_type=None, prev_non_tool_role=None) -%}
+{%- set loop_messages = messages -%}
+{%- set enable_thinking = enable_thinking | default(false) -%}
+{%- set preserve_thinking = preserve_thinking | default(false) -%}
+{{- bos_token -}}
+{#- Handle System/Tool Definitions Block -#}
+{%- if enable_thinking or tools or (messages and messages[0]['role'] in ['system', 'developer']) -%}
+    {{- '<|turn>system\n' -}}
+    {#- Inject Thinking token at the very top of the FIRST system turn -#}
+    {%- if enable_thinking -%}
+        {{- '<|think|>\n' -}}
+        {%- set ns.prev_message_type = 'think' -%}
+    {%- endif -%}
+    {%- if messages and messages[0]['role'] in ['system', 'developer'] -%}
+        {%- if messages[0]['content'] is string -%}
+            {{- messages[0]['content'] | trim -}}
+        {%- elif messages[0]['content'] is sequence -%}
+            {%- for item in messages[0]['content'] -%}
+                {{- item['text'] | trim + ' '-}}
+            {%- endfor -%}
+        {%- endif -%}
+        {%- set loop_messages = messages[1:] -%}
+    {%- endif -%}
+    {%- if tools -%}
+        {%- for tool in tools %}
+            {{- '<|tool>' -}}
+            {{- format_function_declaration(tool) | trim -}}
+            {{- '<tool|>' -}}
+        {%- endfor %}
+        {%- set ns.prev_message_type = 'tool' -%}
+    {%- endif -%}
+    {{- '<turn|>\n' -}}
+{%- endif %}
+
+{#- Pre-scan: find last user message index for reasoning guard -#}
+{%- set ns_turn = namespace(last_user_idx=-1) -%}
+{%- for i in range(loop_messages | length) -%}
+    {%- if loop_messages[i]['role'] == 'user' -%}
+        {%- set ns_turn.last_user_idx = i -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- Loop through messages -#}
+{%- for message in loop_messages -%}
+    {%- if message['role'] != 'tool' -%}
+    {%- set ns.prev_message_type = None -%}
+    {%- set role = 'model' if message['role'] == 'assistant' else message['role'] -%}
+    {#- Detect continuation using tracked state — O(1) instead of O(n) backward scan -#}
+    {%- set continue_same_model_turn = (role == 'model' and ns.prev_non_tool_role == 'assistant') -%}
+    {%- if not continue_same_model_turn -%}
+        {{- '<|turn>' + role + '\n' }}
+
+    {%- endif -%}
+
+    {#- Render reasoning/reasoning_content as thinking channel -#}
+    {%- set thinking_text = message.get('reasoning') or message.get('reasoning_content') -%}
+    {%- set thinking_gate = (loop.index0 > ns_turn.last_user_idx) or (preserve_thinking and message.get('tool_calls')) -%}
+    {%- if thinking_text and thinking_gate -%}
+        {{- '<|channel>thought\n' + thinking_text + '\n<channel|>' -}}
+    {%- endif -%}
+
+            {%- if message.get('tool_calls') -%}
+                {%- for tool_call in message.get('tool_calls') -%}
+                    {%- set function = tool_call['function'] -%}
+                    {{- '<|tool_call>call:' + function['name'] + '{' -}}
+                    {%- if function['arguments'] is mapping -%}
+                        {%- set ns_args = namespace(found_first=false) -%}
+                        {%- for key, value in function['arguments'] | dictsort -%}
+                            {%- if ns_args.found_first %},{% endif -%}
+                            {%- set ns_args.found_first = true -%}
+                            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}
+                        {%- endfor -%}
+                    {%- elif function['arguments'] is none -%}
+                    {%- else -%}
+                        {{- raise_exception(
+                            "chat_template: tool_calls[].function.arguments must be a "
+                            "JSON object (mapping), not a string. Deserialize arguments "
+                            "before passing to the template."
+                        ) -}}
+                    {%- endif -%}
+                    {{- '}<tool_call|>' -}}
+                {%- endfor -%}
+                {%- set ns.prev_message_type = 'tool_call' -%}
+            {%- endif -%}
+
+            {%- set ns_tr_out = namespace(flag=false) -%}
+            {%- if message.get('tool_responses') -%}
+                {#- Legacy: tool_responses embedded on the assistant message (Google/Gemma native) -#}
+                {%- for tool_response in message.get('tool_responses') -%}
+                    {{- format_tool_response_block(tool_response['name'] | default('unknown', true), tool_response['response']) -}}
+                    {%- set ns_tr_out.flag = true -%}
+                    {%- set ns.prev_message_type = 'tool_response' -%}
+                {%- endfor -%}
+            {%- elif message.get('tool_calls') -%}
+                {#- OpenAI Chat Completions: forward-scan consecutive role:tool messages -#}
+                {%- set ns_tool_scan = namespace(stopped=false) -%}
+                {%- for k in range(loop.index0 + 1, loop_messages | length) -%}
+                    {%- if ns_tool_scan.stopped -%}
+                    {%- elif loop_messages[k]['role'] != 'tool' -%}
+                        {%- set ns_tool_scan.stopped = true -%}
+                    {%- else -%}
+                        {%- set follow = loop_messages[k] -%}
+                        {#- Resolve tool_call_id to function name -#}
+                        {%- set ns_tname = namespace(name=follow.get('name') or 'unknown') -%}
+                        {%- for tc in message.get('tool_calls') -%}
+                            {%- if tc.get('id') == follow.get('tool_call_id') -%}
+                                {%- set ns_tname.name = tc['function']['name'] -%}
+                            {%- endif -%}
+                        {%- endfor -%}
+                        {#- Handle content as string or content-parts array -#}
+                        {%- set tool_body = follow.get('content') -%}
+                        {%- if tool_body is string -%}
+                            {{- format_tool_response_block(ns_tname.name, tool_body) -}}
+                        {%- elif tool_body is sequence and tool_body is not string -%}
+                            {%- set ns_txt = namespace(s='') -%}
+                            {%- for part in tool_body -%}
+                                {%- if part.get('type') == 'text' -%}
+                                    {%- set ns_txt.s = ns_txt.s + (part.get('text') | default('')) -%}
+                                {%- endif -%}
+                            {%- endfor -%}
+                            {{- format_tool_response_block(ns_tname.name, ns_txt.s) -}}
+                            {%- for part in tool_body -%}
+                                {%- if part.get('type') in ['image', 'image_url'] -%}
+                                    {{- '<|image|>' -}}
+                                {%- elif part.get('type') in ['audio', 'input_audio'] -%}
+                                    {{- '<|audio|>' -}}
+                                {%- elif part.get('type') == 'video' -%}
+                                    {{- '<|video|>' -}}
+                                {%- endif -%}
+                            {%- endfor -%}
+                        {%- else -%}
+                            {{- format_tool_response_block(ns_tname.name, tool_body) -}}
+                        {%- endif -%}
+                        {%- set ns_tr_out.flag = true -%}
+                        {%- set ns.prev_message_type = 'tool_response' -%}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- endif -%}
+
+            {%- set captured_content -%}
+            {%- if message.get('content') is string -%}
+                {%- if role == 'model' -%}
+                    {{- strip_thinking(message['content']) -}}
+                {%- else -%}
+                    {{- message['content'] | trim -}}
+                {%- endif -%}
+            {%- elif message.get('content') is sequence -%}
+                {%- for item in message['content'] -%}
+                    {%- if item.get('type') == 'text' -%}
+                        {%- if role == 'model' -%}
+                            {{- strip_thinking(item['text']) -}}
+                        {%- else -%}
+                            {{- item['text'] | trim -}}
+                        {%- endif -%}
+                    {%- elif item.get('type') in ['image', 'image_url'] -%}
+                        {{- '<|image|>' -}}
+                    {%- elif item.get('type') in ['audio', 'input_audio'] -%}
+                        {{- '<|audio|>' -}}
+                    {%- elif item.get('type') == 'video' -%}
+                        {{- '<|video|>' -}}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- endif -%}
+            {%- endset -%}
+
+            {{- captured_content -}}
+            {%- set has_content = captured_content | trim | length > 0 -%}
+
+        {#- Forward-scan: find next non-tool message role for continuation detection -#}
+        {%- set next_nt = namespace(role=None, found=false) -%}
+        {%- for j in range(loop.index0 + 1, loop_messages | length) -%}
+            {%- if not next_nt.found -%}
+                {%- if loop_messages[j]['role'] != 'tool' -%}
+                    {%- set next_nt.role = loop_messages[j]['role'] -%}
+                    {%- set next_nt.found = true -%}
+                {%- endif -%}
+            {%- endif -%}
+        {%- endfor -%}
+
+        {%- set continues_into_next = (
+            role == 'model'
+            and next_nt.role == 'assistant'
+            and (not message.get('tool_calls') or ns_tr_out.flag)
+        ) -%}
+
+        {%- if ns.prev_message_type == 'tool_call' and not ns_tr_out.flag -%}
+            {{- '<|tool_response>' -}}
+        {%- elif continues_into_next -%}
+        {%- elif not (ns_tr_out.flag and not has_content and not next_nt.found) -%}
+            {{- '<turn|>\n' -}}
+        {%- endif -%}
+
+    {#- Track previous non-tool role for next iteration (avoids O(n) backward scan) -#}
+    {%- set ns.prev_non_tool_role = message['role'] -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+    {%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
+        {{- '<|turn>model\n' -}}
+        {%- if not enable_thinking -%}
+            {{- '<|channel>thought\n<channel|>' -}}
+        {%- endif -%}
+    {%- elif ns.prev_message_type == 'tool_response' and enable_thinking -%}
+        {{- '<|channel>thought\n' -}}
+    {%- endif -%}
+{%- endif -%}

config.json

@@ -0,0 +1,146 @@
+{
+  "architectures": [
+    "Gemma4ForConditionalGeneration"
+  ],
+  "audio_config": null,
+  "audio_token_id": 258881,
+  "boa_token_id": 256000,
+  "boi_token_id": 255999,
+  "dtype": "bfloat16",
+  "eoa_token_id": 258883,
+  "eoa_token_index": 258883,
+  "eoi_token_id": 258882,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "image_token_id": 258880,
+  "initializer_range": 0.02,
+  "model_type": "gemma4",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attention_k_eq_v": true,
+    "bos_token_id": 2,
+    "dtype": "bfloat16",
+    "enable_moe_block": true,
+    "eos_token_id": 1,
+    "final_logit_softcapping": 30.0,
+    "global_head_dim": 512,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 2816,
+    "hidden_size_per_layer_input": 0,
+    "initializer_range": 0.02,
+    "intermediate_size": 2112,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 262144,
+    "model_type": "gemma4_text",
+    "moe_intermediate_size": 704,
+    "num_attention_heads": 16,
+    "num_experts": 128,
+    "num_global_key_value_heads": 2,
+    "num_hidden_layers": 30,
+    "num_key_value_heads": 8,
+    "num_kv_shared_layers": 0,
+    "pad_token_id": 0,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "partial_rotary_factor": 0.25,
+        "rope_theta": 1000000.0,
+        "rope_type": "proportional"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "tie_word_embeddings": true,
+    "top_k_experts": 8,
+    "use_bidirectional_attention": "vision",
+    "use_cache": true,
+    "use_double_wide_mlp": false,
+    "vocab_size": 262144,
+    "vocab_size_per_layer_input": 262144
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0.dev0",
+  "video_token_id": 258884,
+  "vision_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "chunk_size_feed_forward": 0,
+    "default_output_length": 280,
+    "dtype": "bfloat16",
+    "global_head_dim": 72,
+    "head_dim": 72,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "max_position_embeddings": 131072,
+    "model_type": "gemma4_vision",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27,
+    "num_key_value_heads": 16,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "position_embedding_size": 10240,
+    "problem_type": null,
+    "return_dict": true,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "rope_theta": 100.0,
+      "rope_type": "default"
+    },
+    "standardize": true,
+    "use_clipped_linears": false
+  },
+  "vision_soft_tokens_per_image": 280
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106,
+    50
+  ],
+  "pad_token_id": 0,
+  "temperature": 1.0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "5.5.0.dev0"
+}

head.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ffb37579b962dfb52c3b9684446649635b5a78ca3f7f4210c889421bf5851ad
+size 270584

judge_config.json

@@ -0,0 +1,100 @@
+{
+  "base_model_path": ".",
+  "hidden_size": 2816,
+  "slots": {
+    "num_slots": 24,
+    "ranges": {
+      "noul": [
+        0,
+        2
+      ],
+      "score": [
+        2,
+        8
+      ],
+      "choice": [
+        8,
+        24
+      ]
+    },
+    "verbalizers": [
+      "false",
+      "true",
+      "0",
+      "1",
+      "2",
+      "3",
+      "4",
+      "5",
+      "A",
+      "B",
+      "C",
+      "D",
+      "E",
+      "F",
+      "G",
+      "H",
+      "I",
+      "J",
+      "K",
+      "L",
+      "M",
+      "N",
+      "O",
+      "P"
+    ],
+    "template_version": "bare-v1"
+  },
+  "verbalizer_ids": [
+    4530,
+    3397,
+    236771,
+    236770,
+    236778,
+    236800,
+    236812,
+    236810,
+    236776,
+    236799,
+    236780,
+    236796,
+    236788,
+    236811,
+    236823,
+    236814,
+    236777,
+    236863,
+    236855,
+    236798,
+    236792,
+    236797,
+    236806,
+    236791
+  ],
+  "kinds": [
+    "noul",
+    "choice",
+    "score"
+  ],
+  "softcap": 30.0,
+  "readout": {
+    "type": "last_token",
+    "canvas_len": 1,
+    "canvas_token": 0,
+    "prefix_token": 2,
+    "no_pad_mask": true
+  },
+  "stage": "s2",
+  "lora": {
+    "r": 32,
+    "alpha": 64,
+    "dropout": 0.05,
+    "target_modules": "model\\.language_model\\.layers\\.\\d+\\.(self_attn\\.(q|k|v|o)_proj|mlp\\.(gate|up|down)_proj)"
+  },
+  "model_name": "gev-26b-a4b",
+  "model_version": "0.1.0",
+  "adapter_subfolder": "adapter",
+  "weights_mode": "dual",
+  "base_model": "google/gemma-4-26B-A4B-it",
+  "temperatures": "jev"
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1127684971bbca40465435a5cad69d67ad603bf5e61c6dfd5561fae4a3bcfdb3
+size 49907246508

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aab47033e1e8a492ef8e581efae1cf36478d0433567e7729b3c1728bc8970db7
+size 1704763408

patches/vllm-gemma4-lm-head-lora.patch

@@ -0,0 +1,89 @@
+From f8a48c40b339da4bdfeb86cf8bf179f5e4e397d9 Mon Sep 17 00:00:00 2001
+From: yuhai-china <5901109+yuhai-china@users.noreply.github.com>
+Date: Tue, 29 Sep 2026 16:22:28 +0000
+Subject: [PATCH 1/2] [LoRA] Gemma-4: allow lm_head LoRA (declare
+ embedding_modules; tied lm_head) and raise LoRA vocab limit to 262144
+
+---
+ vllm/lora/layers/logits_processor.py | 6 ++++--
+ vllm/model_executor/models/gemma4.py | 7 +++++++
+ 2 files changed, 11 insertions(+), 2 deletions(-)
+
+diff --git a/vllm/lora/layers/logits_processor.py b/vllm/lora/layers/logits_processor.py
+index 0199437..33b6a2d 100644
+--- a/vllm/lora/layers/logits_processor.py
++++ b/vllm/lora/layers/logits_processor.py
+@@ -88,8 +88,10 @@ class LogitsProcessorWithLoRA(BaseLayerWithLoRA):
+         model_config: PretrainedConfig | None = None,
+     ) -> None:
+         # TODO: Verify if this condition can be further relaxed
+-        if self.base_layer.vocab_size > 258048:
+-            raise ValueError("When using LoRA, vocab size must be <= 258048")
++        # Raised from 258048 to 262144 so that Gemma-4 (vocab 262,144) can carry an
++        # lm_head LoRA; verified by output parity for Gemma-4-26B-A4B.
++        if self.base_layer.vocab_size > 262144:
++            raise ValueError("When using LoRA, vocab size must be <= 262144")
+         self.lora_a_stacked = torch.zeros(
+             (
+                 max_loras,
+diff --git a/vllm/model_executor/models/gemma4.py b/vllm/model_executor/models/gemma4.py
+index 7f850f4..95c29de 100644
+--- a/vllm/model_executor/models/gemma4.py
++++ b/vllm/model_executor/models/gemma4.py
+@@ -1548,6 +1548,13 @@ class Gemma4ForCausalLM(
+             "up_proj",
+         ],
+     }
++    # Maps PEFT lm_head LoRA targets onto vLLM's logits-processor LoRA wrapper.
++    # Gemma-4 ties lm_head to the input embeddings; the LoRA delta is applied to
++    # the output logits only (before the final-logit soft cap), so the input
++    # embeddings and the base model's logits are unchanged when no LoRA is active.
++    embedding_modules = {
++        "lm_head": "output_embeddings",
++    }
+ 
+     def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+         config = _get_text_config(vllm_config.model_config.hf_config)
+-- 
+2.34.1
+
+
+From c7083ba5922c4822fe5a1efb7e9bd0d71c905ec1 Mon Sep 17 00:00:00 2001
+From: yuhai-china <5901109+yuhai-china@users.noreply.github.com>
+Date: Tue, 29 Sep 2026 21:46:44 +0000
+Subject: [PATCH 2/2] [LoRA] Dedupe aliased modules at activation (backport of
+ vllm-project/vllm#39816; fixes Gemma-4 LoRA silently reset)
+
+---
+ vllm/lora/model_manager.py | 13 +++++++++++++
+ 1 file changed, 13 insertions(+)
+
+diff --git a/vllm/lora/model_manager.py b/vllm/lora/model_manager.py
+index d19bf31..8804dcb 100644
+--- a/vllm/lora/model_manager.py
++++ b/vllm/lora/model_manager.py
+@@ -336,8 +336,21 @@ class LoRAModelManager:
+             "Activating LoRA. int id: %d, slot index: %d", lora_model.id, index
+         )
+         self.lora_index_to_id[index] = lora_model.id
++        # Aliased module paths (e.g. Gemma-4 `model.layers.N` and
++        # `model.self_decoder.decoder_layers.N`) point to the same physical LoRA
++        # wrapper: pick one entry per physical module, preferring the path that has
++        # weights, so a later alias cannot reset weights that were just loaded
++        # (upstream vllm-project/vllm#39816, fixes #39815).
++        chosen_modules: dict = {}
+         for module_name, module in self.modules.items():
++            module_key = id(module)
+             module_lora = self._get_lora_layer_weights(lora_model, module_name)
++            if module_key not in chosen_modules:
++                chosen_modules[module_key] = (module_name, module, module_lora)
++            elif module_lora is not None and chosen_modules[module_key][2] is None:
++                chosen_modules[module_key] = (module_name, module, module_lora)
++
++        for module_name, module, module_lora in chosen_modules.values():
+             if not module_lora:
+                 module.reset_lora(index)
+                 logger.debug(
+-- 
+2.34.1
+

processor_config.json

@@ -0,0 +1,75 @@
+{
+  "audio_ms_per_token": 40,
+  "audio_seq_length": 750,
+  "feature_extractor": {
+    "dither": 0.0,
+    "feature_extractor_type": "Gemma4AudioFeatureExtractor",
+    "feature_size": 128,
+    "fft_length": 512,
+    "fft_overdrive": false,
+    "frame_length": 320,
+    "hop_length": 160,
+    "input_scale_factor": 1.0,
+    "max_frequency": 8000.0,
+    "mel_floor": 0.001,
+    "min_frequency": 0.0,
+    "padding_side": "right",
+    "padding_value": 0.0,
+    "per_bin_mean": null,
+    "per_bin_stddev": null,
+    "preemphasis": 0.0,
+    "preemphasis_htk_flavor": true,
+    "return_attention_mask": true,
+    "sampling_rate": 16000
+  },
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": false,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_processor_type": "Gemma4ImageProcessor",
+    "image_seq_length": 280,
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 280,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098
+  },
+  "image_seq_length": 280,
+  "processor_class": "Gemma4Processor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 70,
+    "num_frames": 32,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "video_processor_type": "Gemma4VideoProcessor"
+  }
+}

reports/adaptive_validation_summary.json

@@ -0,0 +1,77 @@
+{
+ "settings": {
+  "threshold": 0.8,
+  "mix": 0.5,
+  "think_budget": 8192
+ },
+ "datasets": {
+  "AQuA-RAT": {
+   "n": 254,
+   "system1": 68.11023622047244,
+   "adaptive": 88.9763779527559,
+   "always_think": 90.15748031496062,
+   "escalated_pct": 53.937007874015755,
+   "ece_system1": 0.03542993923235089,
+   "ece_adaptive": 0.10937879489051755
+  },
+  "CommonsenseQA": {
+   "n": 300,
+   "system1": 86.66666666666667,
+   "adaptive": 85.0,
+   "always_think": 83.33333333333334,
+   "escalated_pct": 32.0,
+   "ece_system1": 0.04021547925836765,
+   "ece_adaptive": 0.053498085540242254
+  },
+  "LogiQA": {
+   "n": 300,
+   "system1": 55.333333333333336,
+   "adaptive": 80.33333333333333,
+   "always_think": 82.0,
+   "escalated_pct": 63.66666666666667,
+   "ece_system1": 0.12319165112240366,
+   "ece_adaptive": 0.12393589949576163
+  },
+  "MedMCQA": {
+   "n": 300,
+   "system1": 66.66666666666666,
+   "adaptive": 71.66666666666667,
+   "always_think": 73.66666666666667,
+   "escalated_pct": 52.33333333333333,
+   "ece_system1": 0.09009478448570694,
+   "ece_adaptive": 0.11489545268113895
+  },
+  "OpenBookQA": {
+   "n": 300,
+   "system1": 94.33333333333334,
+   "adaptive": 96.33333333333334,
+   "always_think": 95.66666666666667,
+   "escalated_pct": 14.666666666666666,
+   "ece_system1": 0.0443799860205645,
+   "ece_adaptive": 0.024977894380833043
+  },
+  "StrategyQA": {
+   "n": 300,
+   "system1": 68.0,
+   "adaptive": 78.66666666666666,
+   "always_think": 79.0,
+   "escalated_pct": 71.0,
+   "ece_system1": 0.02861885470538586,
+   "ece_adaptive": 0.0359552437712111
+  },
+  "ALL": {
+   "n": 1754,
+   "system1": 73.31812998859749,
+   "adaptive": 83.35233751425314,
+   "always_think": 83.80843785632838,
+   "escalated_pct": 47.776510832383124,
+   "ece_system1": 0.03522315989905164,
+   "ece_adaptive": 0.03458824415825564
+  },
+  "thinking_tokens": {
+   "median": 2282.0,
+   "p90": 7573.4,
+   "finished_within_8192": 0.9156214367160775
+  }
+ }
+}

reports/decision_index_benchmark_summary.json

@@ -0,0 +1,1445 @@
+{
+  "engine": "jev-gemma4-step7000",
+  "counts": {
+    "ok": 150759
+  },
+  "successful_request_latency_ms": {
+    "median": 18.467556714313105,
+    "p95": 215.19578801235184,
+    "mean": 72.46806872574511
+  },
+  "benchmarks": [
+    {
+      "catalog_id": 1,
+      "dataset": "BFCL",
+      "requests": 1694,
+      "answered": 1694,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1694,
+      "metric": "case exact accuracy",
+      "score": 0.9462809917355371,
+      "reference_same_cases": null,
+      "median_ms": 138.82611655571964,
+      "detail": {
+        "field_accuracy": 0.9790268456375839,
+        "case_exact_accuracy": 0.9462809917355371,
+        "custom_metrics": {},
+        "positive_micro_f1": 0.9711649365628604
+      }
+    },
+    {
+      "catalog_id": 2,
+      "dataset": "ToolRet",
+      "requests": 685,
+      "answered": 685,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 685,
+      "metric": "nDCG@10",
+      "score": 0.6707486501882342,
+      "reference_same_cases": null,
+      "median_ms": 1291.542490303982,
+      "detail": {
+        "field_accuracy": null,
+        "case_exact_accuracy": null,
+        "custom_metrics": {
+          "ndcg_at_10": 0.6707486501882342,
+          "mrr": 0.7067817063550459,
+          "recall_at_10": 0.7906100104275287,
+          "candidate_recall": 0.8401355578727842,
+          "scorable_candidate_recall": 0.8401355578727842,
+          "candidates_scored": 31.9985401459854,
+          "candidates_retrieved": 32,
+          "bm25_ndcg_at_10": 0.5430661225020075
+        }
+      }
+    },
+    {
+      "catalog_id": 3,
+      "dataset": "API-Bank",
+      "requests": 508,
+      "answered": 508,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 508,
+      "metric": "accuracy",
+      "score": 0.84251968503937,
+      "reference_same_cases": null,
+      "median_ms": 1822.737945985864,
+      "detail": {
+        "field_accuracy": 0.84251968503937,
+        "case_exact_accuracy": 0.84251968503937,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 4,
+      "dataset": "BANKING77",
+      "requests": 3080,
+      "answered": 3080,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 3080,
+      "metric": "macro-F1",
+      "score": 0.8799267700969118,
+      "reference_same_cases": null,
+      "median_ms": 47.48895212833304,
+      "detail": {
+        "field_accuracy": 0.8821428571428571,
+        "case_exact_accuracy": 0.8821428571428571,
+        "macro_f1": 0.8799267700969118,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 5,
+      "dataset": "CLINC150+OOS",
+      "requests": 5500,
+      "answered": 5500,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5500,
+      "metric": "macro-F1",
+      "score": 0.9297695107658135,
+      "reference_same_cases": null,
+      "median_ms": 66.9773818081012,
+      "detail": {
+        "field_accuracy": 0.9089090909090909,
+        "case_exact_accuracy": 0.9089090909090909,
+        "macro_f1": 0.9297695107658135,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 6,
+      "dataset": "RouterBench",
+      "requests": 10000,
+      "answered": 10000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10000,
+      "metric": "selected quality (quality objective)",
+      "score": 0.799375,
+      "reference_same_cases": null,
+      "median_ms": 168.35474733670708,
+      "tracks": {
+        "RouterBench-0shot": {
+          "metric": "selected quality (quality objective)",
+          "score": 0.7865280831501099,
+          "scored_requests": 5003
+        },
+        "RouterBench-5shot": {
+          "metric": "selected quality (quality objective)",
+          "score": 0.8122373424054433,
+          "scored_requests": 4997
+        }
+      },
+      "detail": {
+        "field_accuracy": null,
+        "case_exact_accuracy": null,
+        "custom_metrics": {
+          "quality_quality": 0.799375,
+          "quality_cost_usd": 0.00553126286,
+          "quality_utility": 0.799375,
+          "quality_oracle_optimal": 0.8312,
+          "quality_utility_regret": 0.119685,
+          "cost_aware_quality": 0.779235,
+          "cost_aware_cost_usd": 0.0039796064,
+          "cost_aware_utility": 0.739438936,
+          "cost_aware_oracle_optimal": 0.0314,
+          "cost_aware_utility_regret": 0.176090406628
+        }
+      }
+    },
+    {
+      "catalog_id": 9,
+      "dataset": "Home appliance simulator",
+      "requests": 88,
+      "answered": 88,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 88,
+      "metric": "case exact accuracy",
+      "score": 0.2159090909090909,
+      "reference_same_cases": null,
+      "median_ms": 569.5601852930849,
+      "detail": {
+        "field_accuracy": 0.9345911949685535,
+        "case_exact_accuracy": 0.2159090909090909,
+        "custom_metrics": {},
+        "subgroups": {
+          "outcome": {
+            "n": 88,
+            "mean": 0.8068181818181818
+          },
+          "post_0": {
+            "n": 69,
+            "mean": 0.855072463768116
+          },
+          "post_1": {
+            "n": 25,
+            "mean": 0.84
+          },
+          "resolution": {
+            "n": 88,
+            "mean": 0.7727272727272727
+          },
+          "state_read": {
+            "n": 88,
+            "mean": 1
+          },
+          "target_member": {
+            "n": 1232,
+            "mean": 0.9569805194805194
+          }
+        }
+      }
+    },
+    {
+      "catalog_id": 10,
+      "dataset": "SGD/SGD-X",
+      "requests": 2500,
+      "answered": 2500,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2500,
+      "metric": "macro-F1",
+      "score": 0.32021537489440627,
+      "reference_same_cases": null,
+      "median_ms": 48.37857997335959,
+      "detail": {
+        "field_accuracy": 0.2868,
+        "case_exact_accuracy": 0.2868,
+        "macro_f1": 0.32021537489440627,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 11,
+      "dataset": "ContractNLI",
+      "requests": 123,
+      "answered": 123,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 123,
+      "metric": "macro-F1",
+      "score": 0.8306451780650824,
+      "reference_same_cases": null,
+      "median_ms": 1297.6135823118966,
+      "detail": {
+        "field_accuracy": 0.8689622190339551,
+        "case_exact_accuracy": 0.06504065040650407,
+        "macro_f1": 0.8306451780650824,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 12,
+      "dataset": "ANLI",
+      "requests": 3200,
+      "answered": 3200,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 3200,
+      "metric": "macro-F1",
+      "score": 0.7033787027244137,
+      "reference_same_cases": null,
+      "median_ms": 5.992660371703096,
+      "detail": {
+        "field_accuracy": 0.7021875,
+        "case_exact_accuracy": 0.7021875,
+        "macro_f1": 0.7033787027244137,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 20,
+      "dataset": "BPoMP",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.9026,
+      "reference_same_cases": null,
+      "median_ms": 5.784460125141777,
+      "detail": {
+        "field_accuracy": 0.9026,
+        "case_exact_accuracy": 0.6399506781750924,
+        "custom_metrics": {},
+        "subgroups": {
+          "beam search": {
+            "n": 401,
+            "mean": 0.7880299251870324
+          },
+          "corrupted rhyme": {
+            "n": 376,
+            "mean": 0.9601063829787234
+          },
+          "delayed beam search": {
+            "n": 137,
+            "mean": 0.9416058394160584
+          },
+          "delete any 1 nonrhyming word": {
+            "n": 376,
+            "mean": 0.8297872340425532
+          },
+          "delete any 1 rhyming EOL word": {
+            "n": 376,
+            "mean": 0.976063829787234
+          },
+          "delete any 1 word": {
+            "n": 376,
+            "mean": 0.8617021276595744
+          },
+          "delete any 2 nonrhyming words": {
+            "n": 376,
+            "mean": 0.8670212765957447
+          },
+          "delete any 2 rhyming EOL words": {
+            "n": 376,
+            "mean": 0.9707446808510638
+          },
+          "delete any 2 words": {
+            "n": 376,
+            "mean": 0.9202127659574468
+          },
+          "delete any 3 nonrhyming words": {
+            "n": 376,
+            "mean": 0.8882978723404256
+          },
+          "delete any 3 rhyming words": {
+            "n": 376,
+            "mean": 0.9867021276595744
+          },
+          "delete any 3 words": {
+            "n": 376,
+            "mean": 0.9414893617021277
+          },
+          "shuffled rhyming lines": {
+            "n": 336,
+            "mean": 0.8154761904761905
+          },
+          "shuffled rhyming words": {
+            "n": 366,
+            "mean": 0.912568306010929
+          }
+        }
+      }
+    },
+    {
+      "catalog_id": 21,
+      "dataset": "Humicroedit",
+      "requests": 2628,
+      "answered": 2628,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2628,
+      "metric": "accuracy",
+      "score": 0.630517503805175,
+      "reference_same_cases": null,
+      "median_ms": 2.9752281407127157,
+      "detail": {
+        "field_accuracy": 0.630517503805175,
+        "case_exact_accuracy": 0.630517503805175,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 22,
+      "dataset": "POP909-CL",
+      "requests": 2000,
+      "answered": 2000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2000,
+      "metric": "accuracy",
+      "score": 0.1965,
+      "reference_same_cases": null,
+      "median_ms": 1043.9998766378267,
+      "detail": {
+        "field_accuracy": 0.1965,
+        "case_exact_accuracy": 0.1965,
+        "cluster_macro_accuracy": 0.1902056277056277,
+        "custom_metrics": {},
+        "subgroups": {
+          "A#7": {
+            "n": 1,
+            "mean": 0
+          },
+          "A#M": {
+            "n": 88,
+            "mean": 0.20454545454545456
+          },
+          "A#m": {
+            "n": 43,
+            "mean": 0.023255813953488372
+          },
+          "A#m7": {
+            "n": 7,
+            "mean": 0.42857142857142855
+          },
+          "A#maj7": {
+            "n": 3,
+            "mean": 0.6666666666666666
+          },
+          "A7": {
+            "n": 4,
+            "mean": 0
+          },
+          "AM": {
+            "n": 73,
+            "mean": 0.1095890410958904
+          },
+          "Am": {
+            "n": 69,
+            "mean": 0.5652173913043478
+          },
+          "Am7": {
+            "n": 8,
+            "mean": 0.125
+          },
+          "Amaj7": {
+            "n": 3,
+            "mean": 0.3333333333333333
+          },
+          "B7": {
+            "n": 2,
+            "mean": 0
+          },
+          "BM": {
+            "n": 88,
+            "mean": 0.045454545454545456
+          },
+          "Bdim": {
+            "n": 1,
+            "mean": 0
+          },
+          "Bhalf-dim7": {
+            "n": 1,
+            "mean": 0
+          },
+          "Bm": {
+            "n": 65,
+            "mean": 0.06153846153846154
+          },
+          "Bm7": {
+            "n": 6,
+            "mean": 0.6666666666666666
+          },
+          "Bmaj7": {
+            "n": 3,
+            "mean": 0.3333333333333333
+          },
+          "C#7": {
+            "n": 1,
+            "mean": 1
+          },
+          "C#M": {
+            "n": 78,
+            "mean": 0.21794871794871795
+          },
+          "C#aug / Faug / Aaug": {
+            "n": 1,
+            "mean": 0
+          },
+          "C#dim": {
+            "n": 1,
+            "mean": 0
+          },
+          "C#m": {
+            "n": 51,
+            "mean": 0.3333333333333333
+          },
+          "C#m7": {
+            "n": 8,
+            "mean": 0.875
+          },
+          "C#maj7": {
+            "n": 6,
+            "mean": 0.16666666666666666
+          },
+          "C#sus2 / G#sus4": {
+            "n": 4,
+            "mean": 0
+          },
+          "C#sus4 / F#sus2": {
+            "n": 6,
+            "mean": 0.5
+          },
+          "C7": {
+            "n": 2,
+            "mean": 0
+          },
+          "CM": {
+            "n": 99,
+            "mean": 0.36363636363636365
+          },
+          "Cdim": {
+            "n": 1,
+            "mean": 0
+          },
+          "Cm": {
+            "n": 67,
+            "mean": 0.26865671641791045
+          },
+          "Cm7": {
+            "n": 4,
+            "mean": 0.5
+          },
+          "Cmaj7": {
+            "n": 1,
+            "mean": 0
+          },
+          "Csus2 / Gsus4": {
+            "n": 9,
+            "mean": 0.1111111111111111
+          },
+          "Csus4 / Fsus2": {
+            "n": 6,
+            "mean": 0.16666666666666666
+          },
+          "D#7": {
+            "n": 7,
+            "mean": 0
+          },
+          "D#M": {
+            "n": 86,
+            "mean": 0.26744186046511625
+          },
+          "D#aug / Gaug / Baug": {
+            "n": 1,
+            "mean": 0
+          },
+          "D#dim": {
+            "n": 1,
+            "mean": 0
+          },
+          "D#m": {
+            "n": 43,
+            "mean": 0.20930232558139536
+          },
+          "D#m7": {
+            "n": 3,
+            "mean": 0.3333333333333333
+          },
+          "D#maj7": {
+            "n": 6,
+            "mean": 0
+          },
+          "D#sus2 / A#sus4": {
+            "n": 7,
+            "mean": 0
+          },
+          "D#sus4 / G#sus2": {
+            "n": 4,
+            "mean": 0.25
+          },
+          "D7": {
+            "n": 5,
+            "mean": 0
+          },
+          "DM": {
+            "n": 77,
+            "mean": 0.4155844155844156
+          },
+          "Dm": {
+            "n": 64,
+            "mean": 0.1875
+          },
+          "Dm7": {
+            "n": 8,
+            "mean": 0.375
+          },
+          "DmMaj7": {
+            "n": 1,
+            "mean": 1
+          },
+          "Dmaj7": {
+            "n": 3,
+            "mean": 0
+          },
+          "Dsus2 / Asus4": {
+            "n": 12,
+            "mean": 0.08333333333333333
+          },
+          "Dsus4 / Gsus2": {
+            "n": 5,
+            "mean": 0.2
+          },
+          "E7": {
+            "n": 2,
+            "mean": 0
+          },
+          "EM": {
+            "n": 80,
+            "mean": 0.3
+          },
+          "Edim": {
+            "n": 1,
+            "mean": 0
+          },
+          "Em": {
+            "n": 74,
+            "mean": 0.0945945945945946
+          },
+          "Em7": {
+            "n": 6,
+            "mean": 0.3333333333333333
+          },
+          "Emaj7": {
+            "n": 1,
+            "mean": 0
+          },
+          "Esus2 / Bsus4": {
+            "n": 6,
+            "mean": 0.16666666666666666
+          },
+          "Esus4 / Asus2": {
+            "n": 8,
+            "mean": 0.125
+          },
+          "F#7": {
+            "n": 2,
+            "mean": 0
+          },
+          "F#M": {
+            "n": 62,
+            "mean": 0.0967741935483871
+          },
+          "F#dim": {
+            "n": 1,
+            "mean": 0
+          },
+          "F#half-dim7": {
+            "n": 1,
+            "mean": 0
+          },
+          "F#m": {
+            "n": 58,
+            "mean": 0.1724137931034483
+          },
+          "F#m7": {
+            "n": 9,
+            "mean": 0.4444444444444444
+          },
+          "F#mMaj7": {
+            "n": 1,
+            "mean": 0
+          },
+          "F#maj7": {
+            "n": 4,
+            "mean": 0
+          },
+          "F#sus4 / Bsus2": {
+            "n": 5,
+            "mean": 0
+          },
+          "F7": {
+            "n": 2,
+            "mean": 0
+          },
+          "FM": {
+            "n": 93,
+            "mean": 0.22580645161290322
+          },
+          "Fm": {
+            "n": 55,
+            "mean": 0.16363636363636364
+          },
+          "Fm7": {
+            "n": 7,
+            "mean": 0.42857142857142855
+          },
+          "Fmaj7": {
+            "n": 4,
+            "mean": 0.25
+          },
+          "Fsus4 / A#sus2": {
+            "n": 9,
+            "mean": 0.1111111111111111
+          },
+          "G#7": {
+            "n": 6,
+            "mean": 0.16666666666666666
+          },
+          "G#M": {
+            "n": 88,
+            "mean": 0.03409090909090909
+          },
+          "G#m": {
+            "n": 50,
+            "mean": 0.1
+          },
+          "G#m7": {
+            "n": 4,
+            "mean": 0.5
+          },
+          "G#maj7": {
+            "n": 6,
+            "mean": 0
+          },
+          "G7": {
+            "n": 2,
+            "mean": 0
+          },
+          "GM": {
+            "n": 102,
+            "mean": 0.0196078431372549
+          },
+          "Gdim": {
+            "n": 1,
+            "mean": 0
+          },
+          "Gm": {
+            "n": 66,
+            "mean": 0.030303030303030304
+          },
+          "Gm7": {
+            "n": 7,
+            "mean": 0.7142857142857143
+          },
+          "Gmaj7": {
+            "n": 2,
+            "mean": 0
+          },
+          "NoChord": {
+            "n": 29,
+            "mean": 0.3103448275862069
+          },
+          "Other": {
+            "n": 3,
+            "mean": 0
+          }
+        }
+      }
+    },
+    {
+      "catalog_id": 23,
+      "dataset": "cfcolor",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.6316,
+      "reference_same_cases": null,
+      "median_ms": 14.917532666004263,
+      "detail": {
+        "field_accuracy": 0.6316,
+        "case_exact_accuracy": 0.6316,
+        "cluster_macro_accuracy": 0.6369418544918806,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 24,
+      "dataset": "MMLU",
+      "requests": 14033,
+      "answered": 14033,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 14033,
+      "metric": "accuracy",
+      "score": 0.8351742321670348,
+      "reference_same_cases": null,
+      "median_ms": 5.900340576772578,
+      "detail": {
+        "field_accuracy": 0.8351742321670348,
+        "case_exact_accuracy": 0.8351742321670348,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 25,
+      "dataset": "GPQA Diamond",
+      "requests": 196,
+      "answered": 196,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 196,
+      "metric": "accuracy",
+      "score": 0.4387755102040816,
+      "reference_same_cases": null,
+      "median_ms": 21.98228452471085,
+      "detail": {
+        "field_accuracy": 0.4387755102040816,
+        "case_exact_accuracy": 0.4387755102040816,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 26,
+      "dataset": "ARC-Easy",
+      "requests": 2376,
+      "answered": 2376,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2376,
+      "metric": "accuracy",
+      "score": 0.9861111111111112,
+      "reference_same_cases": null,
+      "median_ms": 4.468785729841329,
+      "detail": {
+        "field_accuracy": 0.9861111111111112,
+        "case_exact_accuracy": 0.9861111111111112,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 27,
+      "dataset": "ARC-Challenge",
+      "requests": 1172,
+      "answered": 1172,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1172,
+      "metric": "accuracy",
+      "score": 0.9658703071672355,
+      "reference_same_cases": null,
+      "median_ms": 5.025674821808934,
+      "detail": {
+        "field_accuracy": 0.9658703071672355,
+        "case_exact_accuracy": 0.9658703071672355,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 28,
+      "dataset": "WinoGrande",
+      "requests": 1267,
+      "answered": 1267,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1267,
+      "metric": "accuracy",
+      "score": 0.9100236779794791,
+      "reference_same_cases": null,
+      "median_ms": 2.5069129187613726,
+      "detail": {
+        "field_accuracy": 0.9100236779794791,
+        "case_exact_accuracy": 0.9100236779794791,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 29,
+      "dataset": "HellaSwag",
+      "requests": 10042,
+      "answered": 10042,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10042,
+      "metric": "accuracy",
+      "score": 0.9643497311292571,
+      "reference_same_cases": null,
+      "median_ms": 9.592797941877507,
+      "detail": {
+        "field_accuracy": 0.9643497311292571,
+        "case_exact_accuracy": 0.9643497311292571,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 30,
+      "dataset": "GSM8K",
+      "requests": 2638,
+      "answered": 2638,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2638,
+      "metric": "accuracy",
+      "score": 0.9757391963608795,
+      "reference_same_cases": null,
+      "median_ms": 8.005504249013029,
+      "tracks": {
+        "GSM8K-10choice": {
+          "metric": "accuracy",
+          "score": 0.9924184988627748,
+          "scored_requests": 1319
+        },
+        "GSM8K-4choice": {
+          "metric": "accuracy",
+          "score": 0.959059893858984,
+          "scored_requests": 1319
+        }
+      },
+      "detail": {
+        "field_accuracy": 0.9757391963608795,
+        "case_exact_accuracy": 0.9529946929492039,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 31,
+      "dataset": "ChessBench",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.2398,
+      "reference_same_cases": null,
+      "median_ms": 54.638300483929925,
+      "detail": {
+        "field_accuracy": 0.2398,
+        "case_exact_accuracy": 0.2398,
+        "custom_metrics": {
+          "value_regret": 0.1481518338751564
+        }
+      }
+    },
+    {
+      "catalog_id": 32,
+      "dataset": "MuSR",
+      "requests": 752,
+      "answered": 752,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 752,
+      "metric": "accuracy",
+      "score": 0.660904255319149,
+      "reference_same_cases": null,
+      "median_ms": 38.64438508753665,
+      "detail": {
+        "field_accuracy": 0.660904255319149,
+        "case_exact_accuracy": 0.660904255319149,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 33,
+      "dataset": "SATA-Bench",
+      "requests": 1650,
+      "answered": 1650,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1650,
+      "metric": "case exact accuracy",
+      "score": 0.34424242424242424,
+      "reference_same_cases": null,
+      "median_ms": 111.17707990342751,
+      "detail": {
+        "field_accuracy": 0.848939872398015,
+        "case_exact_accuracy": 0.34424242424242424,
+        "custom_metrics": {},
+        "positive_micro_f1": 0.8003747232158065
+      }
+    },
+    {
+      "catalog_id": 34,
+      "dataset": "SimpleBench",
+      "requests": 10,
+      "answered": 10,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10,
+      "metric": "accuracy",
+      "score": 0.3,
+      "reference_same_cases": null,
+      "median_ms": 475.1623761985684,
+      "detail": {
+        "field_accuracy": 0.3,
+        "case_exact_accuracy": 0.3,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 36,
+      "dataset": "BRIGHT",
+      "requests": 220,
+      "answered": 220,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 220,
+      "metric": "nDCG@10",
+      "score": 0.4598626258964629,
+      "reference_same_cases": null,
+      "median_ms": 1153.4820798406145,
+      "detail": {
+        "field_accuracy": null,
+        "case_exact_accuracy": null,
+        "custom_metrics": {
+          "ndcg_at_10": 0.4598626258964629,
+          "mrr": 0.6211010141712584,
+          "recall_at_10": 0.4926779384624571,
+          "candidate_recall": 0.5445170600851817,
+          "scorable_candidate_recall": 0.5445170600851817,
+          "candidates_scored": 32,
+          "candidates_retrieved": 32,
+          "bm25_ndcg_at_10": 0.2761244257399229
+        }
+      }
+    },
+    {
+      "catalog_id": 37,
+      "dataset": "Amazon ESCI",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "macro-F1",
+      "score": 0.6082055876720927,
+      "reference_same_cases": null,
+      "median_ms": 28.519113766378723,
+      "detail": {
+        "field_accuracy": 0.7424,
+        "case_exact_accuracy": 0.7424,
+        "macro_f1": 0.6082055876720927,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 38,
+      "dataset": "ACOS",
+      "requests": 1565,
+      "answered": 1565,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1565,
+      "metric": "per-review F1",
+      "score": 0.24538704031549788,
+      "reference_same_cases": null,
+      "median_ms": 239.38009215635248,
+      "detail": {
+        "field_accuracy": 0.9714354718306767,
+        "case_exact_accuracy": 0.0475,
+        "custom_metrics": {},
+        "positive_micro_f1": 0.1858573216520651
+      }
+    },
+    {
+      "catalog_id": 39,
+      "dataset": "FinEntity",
+      "requests": 979,
+      "answered": 979,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 979,
+      "metric": "macro-F1",
+      "score": 0.8893744438777665,
+      "reference_same_cases": null,
+      "median_ms": 12.923071830300614,
+      "detail": {
+        "field_accuracy": 0.8896195396899953,
+        "case_exact_accuracy": 0.8161389172625128,
+        "macro_f1": 0.8893744438777665,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 40,
+      "dataset": "iSarcasmEval",
+      "requests": 4600,
+      "answered": 4600,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 4600,
+      "metric": "see separate subtask tracks",
+      "score": null,
+      "reference_same_cases": null,
+      "median_ms": 3.3754319592844695,
+      "tracks": {
+        "iSarcasmEval-A-Ar": {
+          "metric": "see separate subtask tracks",
+          "score": null,
+          "scored_requests": 1400
+        },
+        "iSarcasmEval-A-En": {
+          "metric": "see separate subtask tracks",
+          "score": null,
+          "scored_requests": 1400
+        },
+        "iSarcasmEval-B-En": {
+          "metric": "see separate subtask tracks",
+          "score": null,
+          "scored_requests": 1400
+        },
+        "iSarcasmEval-C-Ar": {
+          "metric": "see separate subtask tracks",
+          "score": null,
+          "scored_requests": 200
+        },
+        "iSarcasmEval-C-En": {
+          "metric": "see separate subtask tracks",
+          "score": null,
+          "scored_requests": 200
+        }
+      },
+      "detail": {
+        "field_accuracy": 0.9000862068965517,
+        "case_exact_accuracy": 0.8076086956521739,
+        "macro_f1": 0.719505986243066,
+        "custom_metrics": {},
+        "positive_f1_by_field": {
+          "sarcastic": 0.5152439024390244,
+          "sarcasm": 0.5620915032679739,
+          "irony": 0.1037037037037037,
+          "satire": 0.11320754716981132,
+          "understatement": 0.0,
+          "overstatement": 0.04395604395604396,
+          "rhetorical_question": 0.14012738853503184
+        },
+        "category_macro_f1": 0.21119001272451274
+      }
+    },
+    {
+      "catalog_id": 41,
+      "dataset": "VAST",
+      "requests": 3006,
+      "answered": 3006,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 3006,
+      "metric": "macro-F1",
+      "score": 0.6086087530434999,
+      "reference_same_cases": null,
+      "median_ms": 8.309710407047532,
+      "detail": {
+        "field_accuracy": 0.6087824351297405,
+        "case_exact_accuracy": 0.6087824351297405,
+        "macro_f1": 0.6086087530434999,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 42,
+      "dataset": "NLI4CT",
+      "requests": 5500,
+      "answered": 5500,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5500,
+      "metric": "macro-F1",
+      "score": 0.7734587842091947,
+      "reference_same_cases": null,
+      "median_ms": 37.59161247580778,
+      "detail": {
+        "field_accuracy": 0.8001818181818182,
+        "case_exact_accuracy": 0.8001818181818182,
+        "macro_f1": 0.7734587842091947,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 43,
+      "dataset": "CRUXEval",
+      "requests": 570,
+      "answered": 570,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 570,
+      "metric": "accuracy",
+      "score": 0.6719298245614035,
+      "reference_same_cases": null,
+      "median_ms": 11.753883198252879,
+      "detail": {
+        "field_accuracy": 0.6719298245614035,
+        "case_exact_accuracy": 0.6719298245614035,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 44,
+      "dataset": "CLadder",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.716,
+      "reference_same_cases": null,
+      "median_ms": 7.669424390769564,
+      "detail": {
+        "field_accuracy": 0.716,
+        "case_exact_accuracy": 0.716,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 45,
+      "dataset": "HLE",
+      "requests": 501,
+      "answered": 501,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 501,
+      "metric": "accuracy",
+      "score": 0.08582834331337326,
+      "reference_same_cases": null,
+      "median_ms": 38.43188349856064,
+      "detail": {
+        "field_accuracy": 0.08582834331337326,
+        "case_exact_accuracy": 0.08582834331337326,
+        "custom_metrics": {}
+      }
+    },
+    {
+      "catalog_id": 48,
+      "dataset": "ForecastBench",
+      "requests": 10139,
+      "answered": 10139,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10139,
+      "metric": "Brier (lower is better)",
+      "score": 0.1974167741612724,
+      "reference_same_cases": null,
+      "median_ms": 29.15050310548395,
+      "detail": {
+        "field_accuracy": null,
+        "case_exact_accuracy": null,
+        "custom_metrics": {
+          "brier": 0.1974167741612724,
+          "log_loss": 0.5803510075214148,
+          "accuracy": 0.7050004931452806
+        },
+        "subgroups": {
+          "joint": {
+            "n": 3242,
+            "mean": 0.18390476990829577
+          },
+          "scalar": {
+            "n": 6897,
+            "mean": 0.2037682193966139
+          }
+        }
+      }
+    },
+    {
+      "catalog_id": 50,
+      "dataset": "Habermas Machine",
+      "requests": 1676,
+      "answered": 1676,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1676,
+      "metric": "accuracy",
+      "score": 0.6079952267303103,
+      "reference_same_cases": null,
+      "median_ms": 36.67889098869637,
+      "detail": {
+        "field_accuracy": 0.6079952267303103,
+        "case_exact_accuracy": 0.6079952267303103,
+        "custom_metrics": {
+          "normalized_rank_regret": 0.09493834526650756
+        }
+      }
+    },
+    {
+      "catalog_id": 56,
+      "dataset": "PhishNChips phishing decisions",
+      "requests": 2000,
+      "answered": 2000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.789,
+      "median_ms": 71.1229310836643,
+      "scored_requests": 2000,
+      "detail": {
+        "field_accuracy": 0.789,
+        "scored_fields": 2000,
+        "chance_on_rows": 0.5
+      }
+    },
+    {
+      "catalog_id": 57,
+      "dataset": "MMLU-Pro",
+      "requests": 12032,
+      "answered": 12032,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.651845079787234,
+      "median_ms": 15.152119543927256,
+      "scored_requests": 12032,
+      "detail": {
+        "field_accuracy": 0.651845079787234,
+        "scored_fields": 12032,
+        "chance_on_rows": 0.11087694718844986
+      }
+    },
+    {
+      "catalog_id": 58,
+      "dataset": "BBH fixed-option tasks",
+      "requests": 5507,
+      "answered": 5507,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.7517704739422553,
+      "median_ms": 7.826623215805739,
+      "scored_requests": 5507,
+      "detail": {
+        "field_accuracy": 0.7517704739422553,
+        "scored_fields": 5507,
+        "chance_on_rows": 0.31010949680343713
+      }
+    },
+    {
+      "catalog_id": 59,
+      "dataset": "RAGTruth response-level hallucination",
+      "requests": 2700,
+      "answered": 2700,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "F1 on hallucinated class",
+      "score": 0.8413865546218487,
+      "median_ms": 37.95317618642002,
+      "scored_requests": 2700,
+      "detail": {
+        "field_accuracy": 0.8881481481481481,
+        "scored_fields": 2700,
+        "chance_on_rows": 0.41125163541212384
+      }
+    },
+    {
+      "catalog_id": 61,
+      "dataset": "HoVer claim verification",
+      "requests": 4000,
+      "answered": 4000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.87875,
+      "median_ms": 20.43018453696277,
+      "scored_requests": 4000,
+      "detail": {
+        "field_accuracy": 0.87875,
+        "scored_fields": 4000,
+        "chance_on_rows": 0.5
+      }
+    },
+    {
+      "catalog_id": 62,
+      "dataset": "When2Call MCQ",
+      "requests": 3652,
+      "answered": 3652,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.8567907995618839,
+      "median_ms": 44.64486979122739,
+      "scored_requests": 3652,
+      "detail": {
+        "field_accuracy": 0.8567907995618839,
+        "scored_fields": 3652,
+        "chance_on_rows": 0.25
+      }
+    },
+    {
+      "catalog_id": 64,
+      "dataset": "New Yorker caption matching",
+      "requests": 528,
+      "answered": 528,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.8200757575757576,
+      "median_ms": 7.580137331387959,
+      "scored_requests": 528,
+      "detail": {
+        "field_accuracy": 0.8200757575757576,
+        "scored_fields": 528,
+        "chance_on_rows": 0.2
+      }
+    }
+  ],
+  "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage.",
+  "edition": "0.2.1"
+}

reports/decision_index_index.json

@@ -0,0 +1,506 @@
+{
+  "edition": "0.2.1",
+  "panel_id": "decision-index-0.2.1",
+  "index": 58.05,
+  "raw_index": 67.35,
+  "scores": {
+    "balanced_skill": 58.05,
+    "balanced_raw": 67.35,
+    "breadth_skill": 56.98
+  },
+  "areas": [
+    {
+      "id": "knowledge",
+      "label": "Knowledge & Reasoning",
+      "raw": 0.5484,
+      "skill": 0.4304,
+      "coverage": 1.0,
+      "n": 10,
+      "benchmarks": [
+        25,
+        30,
+        31,
+        32,
+        33,
+        43,
+        44,
+        45,
+        57,
+        58
+      ]
+    },
+    {
+      "id": "language",
+      "label": "Language Understanding",
+      "raw": 0.7439,
+      "skill": 0.6358,
+      "coverage": 1.0,
+      "n": 10,
+      "benchmarks": [
+        11,
+        12,
+        28,
+        29,
+        38,
+        39,
+        40,
+        41,
+        42,
+        59
+      ]
+    },
+    {
+      "id": "retrieval",
+      "label": "Retrieval & Classification",
+      "raw": 0.7575,
+      "skill": 0.6788,
+      "coverage": 1.0,
+      "n": 6,
+      "benchmarks": [
+        4,
+        5,
+        36,
+        37,
+        56,
+        61
+      ]
+    },
+    {
+      "id": "tools",
+      "label": "Tools & Automation",
+      "raw": 0.7204,
+      "skill": 0.6973,
+      "coverage": 1.0,
+      "n": 5,
+      "benchmarks": [
+        1,
+        2,
+        3,
+        9,
+        62
+      ]
+    },
+    {
+      "id": "arts",
+      "label": "Arts & Human Taste",
+      "raw": 0.5615,
+      "skill": 0.4147,
+      "coverage": 1.0,
+      "n": 7,
+      "benchmarks": [
+        20,
+        21,
+        22,
+        23,
+        48,
+        50,
+        64
+      ]
+    }
+  ],
+  "benchmarks": {
+    "1": {
+      "raw": 0.9463,
+      "skill": 0.9275,
+      "coverage": 1.0,
+      "random": 0.2592,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "2": {
+      "raw": 0.6707,
+      "skill": 0.6198,
+      "coverage": 1.0,
+      "random": 0.1341,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "3": {
+      "raw": 0.8425,
+      "skill": 0.8395,
+      "coverage": 1.0,
+      "random": 0.0189,
+      "rule": "chance",
+      "in_index": true
+    },
+    "4": {
+      "raw": 0.8799,
+      "skill": 0.8784,
+      "coverage": 1.0,
+      "random": 0.0127,
+      "rule": "chance",
+      "in_index": true
+    },
+    "5": {
+      "raw": 0.9298,
+      "skill": 0.9294,
+      "coverage": 1.0,
+      "random": 0.006,
+      "rule": "chance",
+      "in_index": true
+    },
+    "9": {
+      "raw": 0.2159,
+      "skill": 0.2159,
+      "coverage": 1.0,
+      "random": 0.0,
+      "rule": "chance",
+      "in_index": true
+    },
+    "11": {
+      "raw": 0.8306,
+      "skill": 0.7551,
+      "coverage": 1.0,
+      "random": 0.3085,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "12": {
+      "raw": 0.7034,
+      "skill": 0.5557,
+      "coverage": 1.0,
+      "random": 0.3324,
+      "rule": "chance",
+      "in_index": true
+    },
+    "20": {
+      "raw": 0.9043,
+      "skill": 0.8085,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "21": {
+      "raw": 0.6305,
+      "skill": 0.261,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "22": {
+      "raw": 0.1902,
+      "skill": 0.1839,
+      "coverage": 1.0,
+      "random": 0.0078,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "23": {
+      "raw": 0.6369,
+      "skill": 0.2739,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "25": {
+      "raw": 0.4388,
+      "skill": 0.2517,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "28": {
+      "raw": 0.91,
+      "skill": 0.82,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "chance",
+      "in_index": true
+    },
+    "29": {
+      "raw": 0.9643,
+      "skill": 0.9524,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "chance",
+      "in_index": true
+    },
+    "30": {
+      "raw": 0.9757,
+      "skill": 0.9685,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "track",
+      "in_index": true,
+      "tracks": [
+        {
+          "track": "GSM8K-4choice",
+          "score": 0.9591,
+          "headline": false
+        },
+        {
+          "track": "GSM8K-10choice",
+          "score": 0.9924,
+          "headline": false
+        }
+      ]
+    },
+    "31": {
+      "raw": 0.2398,
+      "skill": 0.172,
+      "coverage": 1.0,
+      "random": 0.0819,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "32": {
+      "raw": 0.6609,
+      "skill": 0.4609,
+      "coverage": 1.0,
+      "random": 0.371,
+      "rule": "chance",
+      "in_index": true
+    },
+    "33": {
+      "raw": 0.3442,
+      "skill": 0.3355,
+      "coverage": 1.0,
+      "random": 0.0131,
+      "rule": "chance",
+      "in_index": true
+    },
+    "36": {
+      "raw": 0.4599,
+      "skill": 0.389,
+      "coverage": 1.0,
+      "random": 0.116,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "37": {
+      "raw": 0.6082,
+      "skill": 0.5086,
+      "coverage": 1.0,
+      "random": 0.2027,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "38": {
+      "raw": 0.2454,
+      "skill": 0.2213,
+      "coverage": 1.0,
+      "random": 0.031,
+      "rule": "chance",
+      "in_index": true
+    },
+    "39": {
+      "raw": 0.8894,
+      "skill": 0.8373,
+      "coverage": 1.0,
+      "random": 0.3201,
+      "rule": "chance",
+      "in_index": true
+    },
+    "40": {
+      "raw": 0.6027,
+      "skill": 0.4888,
+      "coverage": 1.0,
+      "random": 0.2227,
+      "rule": "track",
+      "in_index": true,
+      "tracks": [
+        {
+          "track": "A · Arabic",
+          "score": 0.3986,
+          "headline": false
+        },
+        {
+          "track": "A · English",
+          "score": 0.6027,
+          "headline": true
+        },
+        {
+          "track": "C · Arabic pairs",
+          "score": 0.74,
+          "headline": false
+        },
+        {
+          "track": "C · English pairs",
+          "score": 0.935,
+          "headline": false
+        }
+      ]
+    },
+    "41": {
+      "raw": 0.6086,
+      "skill": 0.4129,
+      "coverage": 1.0,
+      "random": 0.3333,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "42": {
+      "raw": 0.7735,
+      "skill": 0.5593,
+      "coverage": 1.0,
+      "random": 0.486,
+      "rule": "chance",
+      "in_index": true
+    },
+    "43": {
+      "raw": 0.6719,
+      "skill": 0.4795,
+      "coverage": 1.0,
+      "random": 0.3697,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "44": {
+      "raw": 0.716,
+      "skill": 0.432,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "45": {
+      "raw": 0.0858,
+      "skill": 0.0,
+      "coverage": 1.0,
+      "random": 0.1641,
+      "rule": "chance",
+      "in_index": true
+    },
+    "48": {
+      "raw": 0.2104,
+      "skill": 0.2104,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "vs baseline",
+      "in_index": true
+    },
+    "50": {
+      "raw": 0.608,
+      "skill": 0.4311,
+      "coverage": 1.0,
+      "random": 0.311,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "56": {
+      "raw": 0.789,
+      "skill": 0.578,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "chance",
+      "in_index": true
+    },
+    "57": {
+      "raw": 0.6518,
+      "skill": 0.6084,
+      "coverage": 1.0,
+      "random": 0.1109,
+      "rule": "chance",
+      "in_index": true
+    },
+    "58": {
+      "raw": 0.7518,
+      "skill": 0.6402,
+      "coverage": 1.0,
+      "random": 0.3101,
+      "rule": "chance",
+      "in_index": true
+    },
+    "59": {
+      "raw": 0.8414,
+      "skill": 0.6712,
+      "coverage": 1.0,
+      "random": 0.5177,
+      "rule": "chance",
+      "in_index": true
+    },
+    "61": {
+      "raw": 0.8788,
+      "skill": 0.7576,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "chance",
+      "in_index": true
+    },
+    "62": {
+      "raw": 0.8568,
+      "skill": 0.8091,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "chance",
+      "in_index": true
+    },
+    "64": {
+      "raw": 0.8201,
+      "skill": 0.7751,
+      "coverage": 1.0,
+      "random": 0.2,
+      "rule": "chance",
+      "in_index": true
+    },
+    "6": {
+      "raw": 0.7994,
+      "skill": 0.5297,
+      "coverage": 1.0,
+      "random": 0.5735,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "10": {
+      "raw": 0.3202,
+      "skill": 0.0,
+      "coverage": 1.0,
+      "random": 0.399,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "24": {
+      "raw": 0.8352,
+      "skill": 0.7803,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "26": {
+      "raw": 0.9861,
+      "skill": 0.9815,
+      "coverage": 1.0,
+      "random": 0.2502,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "27": {
+      "raw": 0.9659,
+      "skill": 0.9545,
+      "coverage": 1.0,
+      "random": 0.2502,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "34": {
+      "raw": 0.3,
+      "skill": 0.16,
+      "coverage": 1.0,
+      "random": 0.1667,
+      "rule": "shown, not counted",
+      "in_index": false
+    }
+  },
+  "coverage": 1.0,
+  "note": "Decision Index 0.2.1 averages 38 benchmarks in five areas. Arts & Human Taste weighs 10%; the other four share 90% in proportion to the square root of their benchmark count (knowledge 25.8%, language 25.8%, retrieval 20.0%, tools 18.3%). Inside an area, gold ★ benchmarks weigh 1.2 and the rest 1.0; the index is 100 x the weighted mean of the five areas. Each benchmark is chance-corrected first, (score - chance) / (1 - chance) clipped to 0-1, so 0 means random guessing and 100 means perfect. Every score is coverage-adjusted, so an unanswered or unsupported request counts as wrong. ForecastBench enters against its baseline: clip((0.25 - Brier) / 0.25) x coverage, so always predicting 0.5 scores zero. MMLU, ARC-Easy, ARC-Challenge, RouterBench, SGD stay on the board as non-index benchmarks. The six interactive environments are still unrun and stay out. Every entrant on the board has results on all 38 index benchmarks. Point estimates only, no uncertainty intervals yet."
+}

reports/decision_index_score_brief.json

@@ -0,0 +1,49 @@
+{
+  "edition": "0.2.1",
+  "decision_index": 58.05,
+  "scores": {
+    "balanced_skill": 58.05,
+    "balanced_raw": 67.35,
+    "breadth_skill": 56.98
+  },
+  "areas": [
+    {
+      "id": "knowledge",
+      "skill": 0.4304,
+      "raw": 0.5484,
+      "coverage": 1.0,
+      "n": 10
+    },
+    {
+      "id": "language",
+      "skill": 0.6358,
+      "raw": 0.7439,
+      "coverage": 1.0,
+      "n": 10
+    },
+    {
+      "id": "retrieval",
+      "skill": 0.6788,
+      "raw": 0.7575,
+      "coverage": 1.0,
+      "n": 6
+    },
+    {
+      "id": "tools",
+      "skill": 0.6973,
+      "raw": 0.7204,
+      "coverage": 1.0,
+      "n": 5
+    },
+    {
+      "id": "arts",
+      "skill": 0.4147,
+      "raw": 0.5615,
+      "coverage": 1.0,
+      "n": 7
+    }
+  ],
+  "completed": 150317,
+  "complete": true,
+  "out": "/root/di-runs-fast/jev-gemma4-step7000"
+}

reports/decision_index_scores.json

@@ -0,0 +1,1448 @@
+{
+  "engine": "jev-gemma4-step7000",
+  "edition": "0.2.1",
+  "generated_utc": "2026-09-29T10:54:53+00:00",
+  "suite": {
+    "edition": "release-v2.1",
+    "requests": 120340,
+    "scoreable": 119898,
+    "excluded": 442,
+    "added_requests": 30419,
+    "benchmarks": 44,
+    "rows_sha256": "b2b56d6fb636837ca469e689087bdbf373dda8de7638aa2da6793e6eda0792d5",
+    "added_sha256": "7429f3c9cdddb772c1cfc42bb2a45e8516b0032152b746e6929f1c8b52f4ce89"
+  },
+  "completed": 150317,
+  "complete": true,
+  "counts": {
+    "ok": 150317
+  },
+  "latency_ms": {
+    "median": 18.5,
+    "p95": 215.2,
+    "mean": 72.5
+  },
+  "decision_index": 58.05,
+  "raw_index": 67.35,
+  "scores": {
+    "balanced_skill": 58.05,
+    "balanced_raw": 67.35,
+    "breadth_skill": 56.98
+  },
+  "areas": [
+    {
+      "id": "knowledge",
+      "label": "Knowledge & Reasoning",
+      "raw": 0.5484,
+      "skill": 0.4304,
+      "coverage": 1.0,
+      "n": 10,
+      "benchmarks": [
+        25,
+        30,
+        31,
+        32,
+        33,
+        43,
+        44,
+        45,
+        57,
+        58
+      ]
+    },
+    {
+      "id": "language",
+      "label": "Language Understanding",
+      "raw": 0.7439,
+      "skill": 0.6358,
+      "coverage": 1.0,
+      "n": 10,
+      "benchmarks": [
+        11,
+        12,
+        28,
+        29,
+        38,
+        39,
+        40,
+        41,
+        42,
+        59
+      ]
+    },
+    {
+      "id": "retrieval",
+      "label": "Retrieval & Classification",
+      "raw": 0.7575,
+      "skill": 0.6788,
+      "coverage": 1.0,
+      "n": 6,
+      "benchmarks": [
+        4,
+        5,
+        36,
+        37,
+        56,
+        61
+      ]
+    },
+    {
+      "id": "tools",
+      "label": "Tools & Automation",
+      "raw": 0.7204,
+      "skill": 0.6973,
+      "coverage": 1.0,
+      "n": 5,
+      "benchmarks": [
+        1,
+        2,
+        3,
+        9,
+        62
+      ]
+    },
+    {
+      "id": "arts",
+      "label": "Arts & Human Taste",
+      "raw": 0.5615,
+      "skill": 0.4147,
+      "coverage": 1.0,
+      "n": 7,
+      "benchmarks": [
+        20,
+        21,
+        22,
+        23,
+        48,
+        50,
+        64
+      ]
+    }
+  ],
+  "index_benchmarks": {
+    "1": {
+      "raw": 0.9463,
+      "skill": 0.9275,
+      "coverage": 1.0,
+      "random": 0.2592,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "2": {
+      "raw": 0.6707,
+      "skill": 0.6198,
+      "coverage": 1.0,
+      "random": 0.1341,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "3": {
+      "raw": 0.8425,
+      "skill": 0.8395,
+      "coverage": 1.0,
+      "random": 0.0189,
+      "rule": "chance",
+      "in_index": true
+    },
+    "4": {
+      "raw": 0.8799,
+      "skill": 0.8784,
+      "coverage": 1.0,
+      "random": 0.0127,
+      "rule": "chance",
+      "in_index": true
+    },
+    "5": {
+      "raw": 0.9298,
+      "skill": 0.9294,
+      "coverage": 1.0,
+      "random": 0.006,
+      "rule": "chance",
+      "in_index": true
+    },
+    "9": {
+      "raw": 0.2159,
+      "skill": 0.2159,
+      "coverage": 1.0,
+      "random": 0.0,
+      "rule": "chance",
+      "in_index": true
+    },
+    "11": {
+      "raw": 0.8306,
+      "skill": 0.7551,
+      "coverage": 1.0,
+      "random": 0.3085,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "12": {
+      "raw": 0.7034,
+      "skill": 0.5557,
+      "coverage": 1.0,
+      "random": 0.3324,
+      "rule": "chance",
+      "in_index": true
+    },
+    "20": {
+      "raw": 0.9043,
+      "skill": 0.8085,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "21": {
+      "raw": 0.6305,
+      "skill": 0.261,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "22": {
+      "raw": 0.1902,
+      "skill": 0.1839,
+      "coverage": 1.0,
+      "random": 0.0078,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "23": {
+      "raw": 0.6369,
+      "skill": 0.2739,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "25": {
+      "raw": 0.4388,
+      "skill": 0.2517,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "28": {
+      "raw": 0.91,
+      "skill": 0.82,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "chance",
+      "in_index": true
+    },
+    "29": {
+      "raw": 0.9643,
+      "skill": 0.9524,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "chance",
+      "in_index": true
+    },
+    "30": {
+      "raw": 0.9757,
+      "skill": 0.9685,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "track",
+      "in_index": true,
+      "tracks": [
+        {
+          "track": "GSM8K-4choice",
+          "score": 0.9591,
+          "headline": false
+        },
+        {
+          "track": "GSM8K-10choice",
+          "score": 0.9924,
+          "headline": false
+        }
+      ]
+    },
+    "31": {
+      "raw": 0.2398,
+      "skill": 0.172,
+      "coverage": 1.0,
+      "random": 0.0819,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "32": {
+      "raw": 0.6609,
+      "skill": 0.4609,
+      "coverage": 1.0,
+      "random": 0.371,
+      "rule": "chance",
+      "in_index": true
+    },
+    "33": {
+      "raw": 0.3442,
+      "skill": 0.3355,
+      "coverage": 1.0,
+      "random": 0.0131,
+      "rule": "chance",
+      "in_index": true
+    },
+    "36": {
+      "raw": 0.4599,
+      "skill": 0.389,
+      "coverage": 1.0,
+      "random": 0.116,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "37": {
+      "raw": 0.6082,
+      "skill": 0.5086,
+      "coverage": 1.0,
+      "random": 0.2027,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "38": {
+      "raw": 0.2454,
+      "skill": 0.2213,
+      "coverage": 1.0,
+      "random": 0.031,
+      "rule": "chance",
+      "in_index": true
+    },
+    "39": {
+      "raw": 0.8894,
+      "skill": 0.8373,
+      "coverage": 1.0,
+      "random": 0.3201,
+      "rule": "chance",
+      "in_index": true
+    },
+    "40": {
+      "raw": 0.6027,
+      "skill": 0.4888,
+      "coverage": 1.0,
+      "random": 0.2227,
+      "rule": "track",
+      "in_index": true,
+      "tracks": [
+        {
+          "track": "A · Arabic",
+          "score": 0.3986,
+          "headline": false
+        },
+        {
+          "track": "A · English",
+          "score": 0.6027,
+          "headline": true
+        },
+        {
+          "track": "C · Arabic pairs",
+          "score": 0.74,
+          "headline": false
+        },
+        {
+          "track": "C · English pairs",
+          "score": 0.935,
+          "headline": false
+        }
+      ]
+    },
+    "41": {
+      "raw": 0.6086,
+      "skill": 0.4129,
+      "coverage": 1.0,
+      "random": 0.3333,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "42": {
+      "raw": 0.7735,
+      "skill": 0.5593,
+      "coverage": 1.0,
+      "random": 0.486,
+      "rule": "chance",
+      "in_index": true
+    },
+    "43": {
+      "raw": 0.6719,
+      "skill": 0.4795,
+      "coverage": 1.0,
+      "random": 0.3697,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "44": {
+      "raw": 0.716,
+      "skill": 0.432,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "45": {
+      "raw": 0.0858,
+      "skill": 0.0,
+      "coverage": 1.0,
+      "random": 0.1641,
+      "rule": "chance",
+      "in_index": true
+    },
+    "48": {
+      "raw": 0.2104,
+      "skill": 0.2104,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "vs baseline",
+      "in_index": true
+    },
+    "50": {
+      "raw": 0.608,
+      "skill": 0.4311,
+      "coverage": 1.0,
+      "random": 0.311,
+      "rule": "track",
+      "in_index": true,
+      "tracks": []
+    },
+    "56": {
+      "raw": 0.789,
+      "skill": 0.578,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "chance",
+      "in_index": true
+    },
+    "57": {
+      "raw": 0.6518,
+      "skill": 0.6084,
+      "coverage": 1.0,
+      "random": 0.1109,
+      "rule": "chance",
+      "in_index": true
+    },
+    "58": {
+      "raw": 0.7518,
+      "skill": 0.6402,
+      "coverage": 1.0,
+      "random": 0.3101,
+      "rule": "chance",
+      "in_index": true
+    },
+    "59": {
+      "raw": 0.8414,
+      "skill": 0.6712,
+      "coverage": 1.0,
+      "random": 0.5177,
+      "rule": "chance",
+      "in_index": true
+    },
+    "61": {
+      "raw": 0.8788,
+      "skill": 0.7576,
+      "coverage": 1.0,
+      "random": 0.5,
+      "rule": "chance",
+      "in_index": true
+    },
+    "62": {
+      "raw": 0.8568,
+      "skill": 0.8091,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "chance",
+      "in_index": true
+    },
+    "64": {
+      "raw": 0.8201,
+      "skill": 0.7751,
+      "coverage": 1.0,
+      "random": 0.2,
+      "rule": "chance",
+      "in_index": true
+    },
+    "6": {
+      "raw": 0.7994,
+      "skill": 0.5297,
+      "coverage": 1.0,
+      "random": 0.5735,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "10": {
+      "raw": 0.3202,
+      "skill": 0.0,
+      "coverage": 1.0,
+      "random": 0.399,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "24": {
+      "raw": 0.8352,
+      "skill": 0.7803,
+      "coverage": 1.0,
+      "random": 0.25,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "26": {
+      "raw": 0.9861,
+      "skill": 0.9815,
+      "coverage": 1.0,
+      "random": 0.2502,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "27": {
+      "raw": 0.9659,
+      "skill": 0.9545,
+      "coverage": 1.0,
+      "random": 0.2502,
+      "rule": "shown, not counted",
+      "in_index": false
+    },
+    "34": {
+      "raw": 0.3,
+      "skill": 0.16,
+      "coverage": 1.0,
+      "random": 0.1667,
+      "rule": "shown, not counted",
+      "in_index": false
+    }
+  },
+  "benchmarks": {
+    "1": {
+      "catalog_id": 1,
+      "dataset": "BFCL",
+      "requests": 1694,
+      "answered": 1694,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1694,
+      "metric": "case exact accuracy",
+      "score": 0.9463,
+      "reference_same_cases": null,
+      "median_ms": 138.8,
+      "index_raw": 0.9463,
+      "index_skill": 0.9275,
+      "coverage": 1.0,
+      "chance": 0.2592,
+      "in_index": true
+    },
+    "2": {
+      "catalog_id": 2,
+      "dataset": "ToolRet",
+      "requests": 685,
+      "answered": 685,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 685,
+      "metric": "nDCG@10",
+      "score": 0.6707,
+      "reference_same_cases": null,
+      "median_ms": 1291.5,
+      "index_raw": 0.6707,
+      "index_skill": 0.6198,
+      "coverage": 1.0,
+      "chance": 0.1341,
+      "in_index": true
+    },
+    "3": {
+      "catalog_id": 3,
+      "dataset": "API-Bank",
+      "requests": 508,
+      "answered": 508,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 508,
+      "metric": "accuracy",
+      "score": 0.8425,
+      "reference_same_cases": null,
+      "median_ms": 1822.7,
+      "index_raw": 0.8425,
+      "index_skill": 0.8395,
+      "coverage": 1.0,
+      "chance": 0.0189,
+      "in_index": true
+    },
+    "4": {
+      "catalog_id": 4,
+      "dataset": "BANKING77",
+      "requests": 3080,
+      "answered": 3080,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 3080,
+      "metric": "macro-F1",
+      "score": 0.8799,
+      "reference_same_cases": null,
+      "median_ms": 47.5,
+      "index_raw": 0.8799,
+      "index_skill": 0.8784,
+      "coverage": 1.0,
+      "chance": 0.0127,
+      "in_index": true
+    },
+    "5": {
+      "catalog_id": 5,
+      "dataset": "CLINC150+OOS",
+      "requests": 5500,
+      "answered": 5500,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5500,
+      "metric": "macro-F1",
+      "score": 0.9298,
+      "reference_same_cases": null,
+      "median_ms": 67.0,
+      "index_raw": 0.9298,
+      "index_skill": 0.9294,
+      "coverage": 1.0,
+      "chance": 0.006,
+      "in_index": true
+    },
+    "6": {
+      "catalog_id": 6,
+      "dataset": "RouterBench",
+      "requests": 10000,
+      "answered": 10000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10000,
+      "metric": "selected quality (quality objective)",
+      "score": 0.7994,
+      "reference_same_cases": null,
+      "median_ms": 168.4,
+      "tracks": {
+        "RouterBench-0shot": {
+          "metric": "selected quality (quality objective)",
+          "score": 0.7865280831501099,
+          "scored_requests": 5003
+        },
+        "RouterBench-5shot": {
+          "metric": "selected quality (quality objective)",
+          "score": 0.8122373424054433,
+          "scored_requests": 4997
+        }
+      },
+      "index_raw": 0.7994,
+      "index_skill": 0.5297,
+      "coverage": 1.0,
+      "chance": 0.5735,
+      "in_index": false
+    },
+    "9": {
+      "catalog_id": 9,
+      "dataset": "Home appliance simulator",
+      "requests": 88,
+      "answered": 88,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 88,
+      "metric": "case exact accuracy",
+      "score": 0.2159,
+      "reference_same_cases": null,
+      "median_ms": 569.6,
+      "index_raw": 0.2159,
+      "index_skill": 0.2159,
+      "coverage": 1.0,
+      "chance": 0.0,
+      "in_index": true
+    },
+    "10": {
+      "catalog_id": 10,
+      "dataset": "SGD/SGD-X",
+      "requests": 2500,
+      "answered": 2500,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2500,
+      "metric": "macro-F1",
+      "score": 0.3202,
+      "reference_same_cases": null,
+      "median_ms": 48.4,
+      "index_raw": 0.3202,
+      "index_skill": 0.0,
+      "coverage": 1.0,
+      "chance": 0.399,
+      "in_index": false
+    },
+    "11": {
+      "catalog_id": 11,
+      "dataset": "ContractNLI",
+      "requests": 123,
+      "answered": 123,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 123,
+      "metric": "macro-F1",
+      "score": 0.8306,
+      "reference_same_cases": null,
+      "median_ms": 1297.6,
+      "index_raw": 0.8306,
+      "index_skill": 0.7551,
+      "coverage": 1.0,
+      "chance": 0.3085,
+      "in_index": true
+    },
+    "12": {
+      "catalog_id": 12,
+      "dataset": "ANLI",
+      "requests": 3200,
+      "answered": 3200,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 3200,
+      "metric": "macro-F1",
+      "score": 0.7034,
+      "reference_same_cases": null,
+      "median_ms": 6.0,
+      "index_raw": 0.7034,
+      "index_skill": 0.5557,
+      "coverage": 1.0,
+      "chance": 0.3324,
+      "in_index": true
+    },
+    "20": {
+      "catalog_id": 20,
+      "dataset": "BPoMP",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.9026,
+      "reference_same_cases": null,
+      "median_ms": 5.8,
+      "index_raw": 0.9043,
+      "index_skill": 0.8085,
+      "coverage": 1.0,
+      "chance": 0.5,
+      "in_index": true
+    },
+    "21": {
+      "catalog_id": 21,
+      "dataset": "Humicroedit",
+      "requests": 2628,
+      "answered": 2628,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2628,
+      "metric": "accuracy",
+      "score": 0.6305,
+      "reference_same_cases": null,
+      "median_ms": 3.0,
+      "index_raw": 0.6305,
+      "index_skill": 0.261,
+      "coverage": 1.0,
+      "chance": 0.5,
+      "in_index": true
+    },
+    "22": {
+      "catalog_id": 22,
+      "dataset": "POP909-CL",
+      "requests": 2000,
+      "answered": 2000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2000,
+      "metric": "accuracy",
+      "score": 0.1965,
+      "reference_same_cases": null,
+      "median_ms": 1044.0,
+      "index_raw": 0.1902,
+      "index_skill": 0.1839,
+      "coverage": 1.0,
+      "chance": 0.0078,
+      "in_index": true
+    },
+    "23": {
+      "catalog_id": 23,
+      "dataset": "cfcolor",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.6316,
+      "reference_same_cases": null,
+      "median_ms": 14.9,
+      "index_raw": 0.6369,
+      "index_skill": 0.2739,
+      "coverage": 1.0,
+      "chance": 0.5,
+      "in_index": true
+    },
+    "24": {
+      "catalog_id": 24,
+      "dataset": "MMLU",
+      "requests": 14033,
+      "answered": 14033,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 14033,
+      "metric": "accuracy",
+      "score": 0.8352,
+      "reference_same_cases": null,
+      "median_ms": 5.9,
+      "index_raw": 0.8352,
+      "index_skill": 0.7803,
+      "coverage": 1.0,
+      "chance": 0.25,
+      "in_index": false
+    },
+    "25": {
+      "catalog_id": 25,
+      "dataset": "GPQA Diamond",
+      "requests": 196,
+      "answered": 196,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 196,
+      "metric": "accuracy",
+      "score": 0.4388,
+      "reference_same_cases": null,
+      "median_ms": 22.0,
+      "index_raw": 0.4388,
+      "index_skill": 0.2517,
+      "coverage": 1.0,
+      "chance": 0.25,
+      "in_index": true
+    },
+    "26": {
+      "catalog_id": 26,
+      "dataset": "ARC-Easy",
+      "requests": 2376,
+      "answered": 2376,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2376,
+      "metric": "accuracy",
+      "score": 0.9861,
+      "reference_same_cases": null,
+      "median_ms": 4.5,
+      "index_raw": 0.9861,
+      "index_skill": 0.9815,
+      "coverage": 1.0,
+      "chance": 0.2502,
+      "in_index": false
+    },
+    "27": {
+      "catalog_id": 27,
+      "dataset": "ARC-Challenge",
+      "requests": 1172,
+      "answered": 1172,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1172,
+      "metric": "accuracy",
+      "score": 0.9659,
+      "reference_same_cases": null,
+      "median_ms": 5.0,
+      "index_raw": 0.9659,
+      "index_skill": 0.9545,
+      "coverage": 1.0,
+      "chance": 0.2502,
+      "in_index": false
+    },
+    "28": {
+      "catalog_id": 28,
+      "dataset": "WinoGrande",
+      "requests": 1267,
+      "answered": 1267,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1267,
+      "metric": "accuracy",
+      "score": 0.91,
+      "reference_same_cases": null,
+      "median_ms": 2.5,
+      "index_raw": 0.91,
+      "index_skill": 0.82,
+      "coverage": 1.0,
+      "chance": 0.5,
+      "in_index": true
+    },
+    "29": {
+      "catalog_id": 29,
+      "dataset": "HellaSwag",
+      "requests": 10042,
+      "answered": 10042,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10042,
+      "metric": "accuracy",
+      "score": 0.9643,
+      "reference_same_cases": null,
+      "median_ms": 9.6,
+      "index_raw": 0.9643,
+      "index_skill": 0.9524,
+      "coverage": 1.0,
+      "chance": 0.25,
+      "in_index": true
+    },
+    "30": {
+      "catalog_id": 30,
+      "dataset": "GSM8K",
+      "requests": 2638,
+      "answered": 2638,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 2638,
+      "metric": "accuracy",
+      "score": 0.9757,
+      "reference_same_cases": null,
+      "median_ms": 8.0,
+      "tracks": [
+        {
+          "track": "GSM8K-4choice",
+          "score": 0.9591,
+          "headline": false
+        },
+        {
+          "track": "GSM8K-10choice",
+          "score": 0.9924,
+          "headline": false
+        }
+      ],
+      "index_raw": 0.9757,
+      "index_skill": 0.9685,
+      "coverage": 1.0,
+      "chance": 0.25,
+      "in_index": true
+    },
+    "31": {
+      "catalog_id": 31,
+      "dataset": "ChessBench",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.2398,
+      "reference_same_cases": null,
+      "median_ms": 54.6,
+      "index_raw": 0.2398,
+      "index_skill": 0.172,
+      "coverage": 1.0,
+      "chance": 0.0819,
+      "in_index": true
+    },
+    "32": {
+      "catalog_id": 32,
+      "dataset": "MuSR",
+      "requests": 752,
+      "answered": 752,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 752,
+      "metric": "accuracy",
+      "score": 0.6609,
+      "reference_same_cases": null,
+      "median_ms": 38.6,
+      "index_raw": 0.6609,
+      "index_skill": 0.4609,
+      "coverage": 1.0,
+      "chance": 0.371,
+      "in_index": true
+    },
+    "33": {
+      "catalog_id": 33,
+      "dataset": "SATA-Bench",
+      "requests": 1650,
+      "answered": 1650,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1650,
+      "metric": "case exact accuracy",
+      "score": 0.3442,
+      "reference_same_cases": null,
+      "median_ms": 111.2,
+      "index_raw": 0.3442,
+      "index_skill": 0.3355,
+      "coverage": 1.0,
+      "chance": 0.0131,
+      "in_index": true
+    },
+    "34": {
+      "catalog_id": 34,
+      "dataset": "SimpleBench",
+      "requests": 10,
+      "answered": 10,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10,
+      "metric": "accuracy",
+      "score": 0.3,
+      "reference_same_cases": null,
+      "median_ms": 475.2,
+      "index_raw": 0.3,
+      "index_skill": 0.16,
+      "coverage": 1.0,
+      "chance": 0.1667,
+      "in_index": false
+    },
+    "36": {
+      "catalog_id": 36,
+      "dataset": "BRIGHT",
+      "requests": 220,
+      "answered": 220,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 220,
+      "metric": "nDCG@10",
+      "score": 0.4599,
+      "reference_same_cases": null,
+      "median_ms": 1153.5,
+      "index_raw": 0.4599,
+      "index_skill": 0.389,
+      "coverage": 1.0,
+      "chance": 0.116,
+      "in_index": true
+    },
+    "37": {
+      "catalog_id": 37,
+      "dataset": "Amazon ESCI",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "macro-F1",
+      "score": 0.6082,
+      "reference_same_cases": null,
+      "median_ms": 28.5,
+      "index_raw": 0.6082,
+      "index_skill": 0.5086,
+      "coverage": 1.0,
+      "chance": 0.2027,
+      "in_index": true
+    },
+    "38": {
+      "catalog_id": 38,
+      "dataset": "ACOS",
+      "requests": 1565,
+      "answered": 1565,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1565,
+      "metric": "per-review F1",
+      "score": 0.2454,
+      "reference_same_cases": null,
+      "median_ms": 239.4,
+      "index_raw": 0.2454,
+      "index_skill": 0.2213,
+      "coverage": 1.0,
+      "chance": 0.031,
+      "in_index": true
+    },
+    "39": {
+      "catalog_id": 39,
+      "dataset": "FinEntity",
+      "requests": 979,
+      "answered": 979,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 979,
+      "metric": "macro-F1",
+      "score": 0.8894,
+      "reference_same_cases": null,
+      "median_ms": 12.9,
+      "index_raw": 0.8894,
+      "index_skill": 0.8373,
+      "coverage": 1.0,
+      "chance": 0.3201,
+      "in_index": true
+    },
+    "40": {
+      "catalog_id": 40,
+      "dataset": "iSarcasmEval",
+      "requests": 4600,
+      "answered": 4600,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 4600,
+      "metric": "Sarcasm F1 · track A, English",
+      "score": 0.6027,
+      "reference_same_cases": null,
+      "median_ms": 3.4,
+      "tracks": [
+        {
+          "track": "A · Arabic",
+          "score": 0.3986,
+          "headline": false
+        },
+        {
+          "track": "A · English",
+          "score": 0.6027,
+          "headline": true
+        },
+        {
+          "track": "C · Arabic pairs",
+          "score": 0.74,
+          "headline": false
+        },
+        {
+          "track": "C · English pairs",
+          "score": 0.935,
+          "headline": false
+        }
+      ],
+      "index_raw": 0.6027,
+      "index_skill": 0.4888,
+      "coverage": 1.0,
+      "chance": 0.2227,
+      "in_index": true
+    },
+    "41": {
+      "catalog_id": 41,
+      "dataset": "VAST",
+      "requests": 3006,
+      "answered": 3006,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 3006,
+      "metric": "macro-F1",
+      "score": 0.6086,
+      "reference_same_cases": null,
+      "median_ms": 8.3,
+      "index_raw": 0.6086,
+      "index_skill": 0.4129,
+      "coverage": 1.0,
+      "chance": 0.3333,
+      "in_index": true
+    },
+    "42": {
+      "catalog_id": 42,
+      "dataset": "NLI4CT",
+      "requests": 5500,
+      "answered": 5500,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5500,
+      "metric": "macro-F1",
+      "score": 0.7735,
+      "reference_same_cases": null,
+      "median_ms": 37.6,
+      "index_raw": 0.7735,
+      "index_skill": 0.5593,
+      "coverage": 1.0,
+      "chance": 0.486,
+      "in_index": true
+    },
+    "43": {
+      "catalog_id": 43,
+      "dataset": "CRUXEval",
+      "requests": 570,
+      "answered": 570,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 570,
+      "metric": "accuracy",
+      "score": 0.6719,
+      "reference_same_cases": null,
+      "median_ms": 11.8,
+      "index_raw": 0.6719,
+      "index_skill": 0.4795,
+      "coverage": 1.0,
+      "chance": 0.3697,
+      "in_index": true
+    },
+    "44": {
+      "catalog_id": 44,
+      "dataset": "CLadder",
+      "requests": 5000,
+      "answered": 5000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 5000,
+      "metric": "accuracy",
+      "score": 0.716,
+      "reference_same_cases": null,
+      "median_ms": 7.7,
+      "index_raw": 0.716,
+      "index_skill": 0.432,
+      "coverage": 1.0,
+      "chance": 0.5,
+      "in_index": true
+    },
+    "45": {
+      "catalog_id": 45,
+      "dataset": "HLE",
+      "requests": 501,
+      "answered": 501,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 501,
+      "metric": "accuracy",
+      "score": 0.0858,
+      "reference_same_cases": null,
+      "median_ms": 38.4,
+      "index_raw": 0.0858,
+      "index_skill": 0.0,
+      "coverage": 1.0,
+      "chance": 0.1641,
+      "in_index": true
+    },
+    "48": {
+      "catalog_id": 48,
+      "dataset": "ForecastBench",
+      "requests": 10139,
+      "answered": 10139,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 10139,
+      "metric": "Brier (lower is better)",
+      "score": 0.1974,
+      "reference_same_cases": null,
+      "median_ms": 29.2,
+      "index_raw": 0.2104,
+      "index_skill": 0.2104,
+      "coverage": 1.0,
+      "chance": 0.25,
+      "in_index": true
+    },
+    "50": {
+      "catalog_id": 50,
+      "dataset": "Habermas Machine",
+      "requests": 1676,
+      "answered": 1676,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "scored_requests": 1676,
+      "metric": "accuracy",
+      "score": 0.608,
+      "reference_same_cases": null,
+      "median_ms": 36.7,
+      "index_raw": 0.608,
+      "index_skill": 0.4311,
+      "coverage": 1.0,
+      "chance": 0.311,
+      "in_index": true
+    },
+    "56": {
+      "catalog_id": 56,
+      "dataset": "PhishNChips phishing decisions",
+      "requests": 2000,
+      "answered": 2000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.789,
+      "median_ms": 71.1,
+      "scored_requests": 2000,
+      "index_raw": 0.789,
+      "index_skill": 0.578,
+      "coverage": 1.0,
+      "chance": 0.5,
+      "in_index": true
+    },
+    "57": {
+      "catalog_id": 57,
+      "dataset": "MMLU-Pro",
+      "requests": 12032,
+      "answered": 12032,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.6518,
+      "median_ms": 15.2,
+      "scored_requests": 12032,
+      "index_raw": 0.6518,
+      "index_skill": 0.6084,
+      "coverage": 1.0,
+      "chance": 0.1109,
+      "in_index": true
+    },
+    "58": {
+      "catalog_id": 58,
+      "dataset": "BBH fixed-option tasks",
+      "requests": 5507,
+      "answered": 5507,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.7518,
+      "median_ms": 7.8,
+      "scored_requests": 5507,
+      "index_raw": 0.7518,
+      "index_skill": 0.6402,
+      "coverage": 1.0,
+      "chance": 0.3101,
+      "in_index": true
+    },
+    "59": {
+      "catalog_id": 59,
+      "dataset": "RAGTruth response-level hallucination",
+      "requests": 2700,
+      "answered": 2700,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "F1 on hallucinated class",
+      "score": 0.8414,
+      "median_ms": 38.0,
+      "scored_requests": 2700,
+      "index_raw": 0.8414,
+      "index_skill": 0.6712,
+      "coverage": 1.0,
+      "chance": 0.5177,
+      "in_index": true
+    },
+    "61": {
+      "catalog_id": 61,
+      "dataset": "HoVer claim verification",
+      "requests": 4000,
+      "answered": 4000,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.8788,
+      "median_ms": 20.4,
+      "scored_requests": 4000,
+      "index_raw": 0.8788,
+      "index_skill": 0.7576,
+      "coverage": 1.0,
+      "chance": 0.5,
+      "in_index": true
+    },
+    "62": {
+      "catalog_id": 62,
+      "dataset": "When2Call MCQ",
+      "requests": 3652,
+      "answered": 3652,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.8568,
+      "median_ms": 44.6,
+      "scored_requests": 3652,
+      "index_raw": 0.8568,
+      "index_skill": 0.8091,
+      "coverage": 1.0,
+      "chance": 0.25,
+      "in_index": true
+    },
+    "64": {
+      "catalog_id": 64,
+      "dataset": "New Yorker caption matching",
+      "requests": 528,
+      "answered": 528,
+      "unsupported": 0,
+      "errors": 0,
+      "abstained": 0,
+      "pending": 0,
+      "metric": "accuracy",
+      "score": 0.8201,
+      "median_ms": 7.6,
+      "scored_requests": 528,
+      "index_raw": 0.8201,
+      "index_skill": 0.7751,
+      "coverage": 1.0,
+      "chance": 0.2,
+      "in_index": true
+    }
+  },
+  "panel_id": "decision-index-0.2.1",
+  "note": "Decision Index 0.2.1 averages 38 benchmarks in five areas. Arts & Human Taste weighs 10%; the other four share 90% in proportion to the square root of their benchmark count (knowledge 25.8%, language 25.8%, retrieval 20.0%, tools 18.3%). Inside an area, gold ★ benchmarks weigh 1.2 and the rest 1.0; the index is 100 x the weighted mean of the five areas. Each benchmark is chance-corrected first, (score - chance) / (1 - chance) clipped to 0-1, so 0 means random guessing and 100 means perfect. Every score is coverage-adjusted, so an unanswered or unsupported request counts as wrong. ForecastBench enters against its baseline: clip((0.25 - Brier) / 0.25) x coverage, so always predicting 0.5 scores zero. MMLU, ARC-Easy, ARC-Challenge, RouterBench, SGD stay on the board as non-index benchmarks. The six interactive environments are still unrun and stay out. Every entrant on the board has results on all 38 index benchmarks. Point estimates only, no uncertainty intervals yet."
+}

serve.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# Serve autotrust/GEV-26B-Decide with vLLM: one engine, both systems, text and images.
+#   System 2: the unmodified gemma-4-26B-A4B-it (served name autotrust/GEV-26B-Decide) on the OpenAI endpoints
+#   System 1: POST /v1/decide (2-256 options, optional adaptive thinking), or the LoRA module "jev-decision" directly
+# serve_decide.py is the standard vLLM OpenAI server (same flags) with /v1/decide added.
+# Requirements: a vLLM build with Gemma-4 support plus patches/vllm-gemma4-lm-head-lora.patch (LoRA on Gemma-4's tied
+# lm_head, vocabulary 262,144), tested with a vLLM development build from September 2026.
+# MAX_MODEL_LEN: up to 262144 (the backbone's native context).
+set -e
+MODEL_DIR=${MODEL_DIR:-GEV-26B-Decide}
+[ -d "$MODEL_DIR" ] || hf download autotrust/GEV-26B-Decide --local-dir "$MODEL_DIR"
+exec python3 "$MODEL_DIR/serve_decide.py" --model "$MODEL_DIR" --served-model-name autotrust/GEV-26B-Decide \
+  --enable-lora --max-lora-rank 32 --lora-modules jev-decision="$MODEL_DIR/adapter_vllm" \
+  --logprobs-mode processed_logprobs --max-model-len ${MAX_MODEL_LEN:-65536} --enable-prefix-caching \
+  --max-num-seqs 256 --trust-request-chat-template --limit-mm-per-prompt '{"image": 8}' \
+  --port ${PORT:-8000}

serve_decide.py

@@ -0,0 +1,367 @@
+#!/usr/bin/env python3
+"""vLLM OpenAI-compatible server with a built-in System 1 endpoint (POST /v1/decide) and adaptive thinking.
+
+One engine serves both systems, for JEV-27B / JEV-27B-VL (Qwen3.8) and GEV-26B-A4B (Gemma-4):
+  System 2  the unmodified base model, through the usual OpenAI endpoints (served model name)
+  System 1  the LoRA module "jev-decision" (backbone LoRA + decision head as an lm_head LoRA), through /v1/decide
+
+POST /v1/decide
+  kind       "noul" (yes/no), "score" (0-5) or "choice" (2-256 options)
+  state      string, JSON object, or a list mixing text and images: ["text", {"image": "https://... | data:..."}]
+  question   string
+  options    list of strings (choice only)
+  strategy   choices with more than 16 options: "single" (one pass, labels A-P then Q-Z, AA, ...), "tournament"
+             (groups of <=16 + a final of 16), "permute" (single pass over 4 option orders, averaged); default per model
+  thinking   "off" (default), "auto" (think only when the leading option is below `threshold`), "on" (always think)
+  threshold  System 1 confidence below which "auto" switches thinking on (default per model)
+  think_budget        maximum thinking tokens (default 8192)
+  return_reasoning    include System 2's reasoning text
+  debug      include the System 1 and System 2 distributions
+
+Adaptive thinking: System 1 gives p1 in one pass. If thinking is switched on, the base model reasons in its thinking
+mode over the same state, question and options; when the thinking channel closes, the answer-letter distribution p2 is
+read in one step, and the result is p = (1 - w) * p1 + w * p2 (default w = 0.5: the fast decision and the reasoning get
+equal weight, which keeps the calibration of System 1; p2 alone is close to one-hot and over-confident).
+
+Response (same shape as the hosted JEV API, plus "thinking" when requested):
+  {"kind", "effective_kind", "options", "probabilities", "choice_index", "choice", "adaptation", "protocol", "model",
+   "usage", "elapsed_seconds", "num_model_requests", "thinking": {...}}
+
+Run it like `python -m vllm.entrypoints.openai.api_server` (same flags), with --lora-modules jev-decision=<adapter dir>
+and --trust-request-chat-template. decision_head.json is read from the adapter directory, calibration.json from its
+parent (override with JEV_DECIDE_CALIBRATION).
+"""
+import asyncio
+import json
+import math
+import os
+import string
+import time
+import zlib
+from typing import Literal
+
+from fastapi import APIRouter, Request
+from fastapi.responses import JSONResponse
+from pydantic import BaseModel
+
+import vllm.entrypoints.launchers.api_server.entry as entry
+from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
+from vllm.entrypoints.openai.completion.protocol import CompletionRequest
+
+LORA = os.environ.get("JEV_DECIDE_LORA", "jev-decision")
+MAX_OPTIONS = 256
+CHUNK = 128  # vLLM caps logprob_token_ids at 128 per request
+PROTOCOL = "jev27-bare-v1"
+PLACEHOLDER = "XQXCONTENTXQX"
+
+# Per-model settings. threshold / mix: adaptive-thinking defaults. GEV: fitted on 1,754 questions outside the Decision
+# Index (CommonsenseQA, OpenBookQA, AQuA-RAT, MedMCQA, LogiQA, StrategyQA). JEV-27B: same defaults, not yet validated.
+PROFILES = {
+    "qwen": {"prefix": "", "image": "<|vision_start|><|image_pad|><|vision_end|>", "end_think": "</think>",
+             "after_think": "\n\n", "strategy": "single", "threshold": 0.8, "mix": 0.5},
+    "gemma": {"prefix": "<bos>", "image": "<|image|>", "end_think": "<channel|>", "after_think": "",
+              "strategy": "tournament", "threshold": 0.8, "mix": 0.5},
+}
+# Read the full distribution: override generation_config defaults (top_k/top_p) that would truncate processed logprobs.
+READ = dict(max_tokens=1, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, repetition_penalty=1.0,
+            add_special_tokens=False, return_tokens_as_token_ids=True)
+S = {}
+
+
+class DecideRequest(BaseModel):
+    kind: Literal["noul", "score", "choice"]
+    state: str | dict | list = ""
+    question: str
+    options: list[str] | None = None
+    strategy: Literal["auto", "single", "tournament", "permute"] = "auto"
+    thinking: Literal["off", "auto", "on"] = "off"
+    threshold: float | None = None
+    think_budget: int = 8192
+    return_reasoning: bool = False
+    debug: bool = False
+
+
+# ------------------------------------------------------------------ setup
+def setup(args):
+    from transformers import AutoConfig, AutoTokenizer
+
+    path = next((m.path for m in (args.lora_modules or []) if m.name == LORA), None)
+    if path is None:
+        raise SystemExit(f"serve_decide: start vLLM with --lora-modules {LORA}=<adapter dir>")
+    head = json.load(open(os.path.join(path, "decision_head.json")))
+    calib = os.environ.get("JEV_DECIDE_CALIBRATION") or os.path.join(os.path.dirname(os.path.abspath(path)), "calibration.json")
+    temps = json.load(open(calib))["per_kind"] if os.path.exists(calib) else {"noul": 1.0, "score": 1.0, "choice": 1.0}
+    temps.setdefault("score", 1.0)
+    src = args.tokenizer or args.model
+    tok = AutoTokenizer.from_pretrained(src, trust_remote_code=args.trust_remote_code)
+    mtype = AutoConfig.from_pretrained(args.model, trust_remote_code=args.trust_remote_code).model_type
+    prof = dict(PROFILES["gemma" if "gemma" in mtype else "qwen"])
+    for k in ("strategy", "threshold", "mix"):
+        if os.environ.get(f"JEV_DECIDE_{k.upper()}"):
+            prof[k] = os.environ[f"JEV_DECIDE_{k.upper()}"] if k == "strategy" else float(os.environ[f"JEV_DECIDE_{k.upper()}"])
+
+    def single_token_labels(context):
+        out = []
+        for lab in list(string.ascii_uppercase) + [a + b for a in string.ascii_uppercase for b in string.ascii_uppercase]:
+            t = tok.encode(lab, add_special_tokens=False)
+            if len(t) == 1 and t[0] in tok.encode(context.format(lab), add_special_tokens=False):
+                out.append((lab, t[0]))
+            if len(out) == MAX_OPTIONS:
+                break
+        return out
+
+    dec = single_token_labels("x\n{}) y")                      # System 1 option-line labels
+    lo, hi = head["slots"]["ranges"]["choice"]
+    assert [t for _, t in dec[: hi - lo]] == head["verbalizer_ids"][lo:hi], "first labels must be the trained A-P head"
+    base = tok.encode("Answer: (", add_special_tokens=False)   # System 2 answer labels
+    ans = []
+    for lab in list(string.ascii_uppercase) + [a + b for a in string.ascii_uppercase for b in string.ascii_uppercase]:
+        ids = tok.encode(f"Answer: ({lab})", add_special_tokens=False)
+        if ids[: len(base)] == base and len(ids) == len(base) + 2:
+            ans.append((lab, ids[len(base)]))
+        if len(ans) == MAX_OPTIONS:
+            break
+    chat = tok.apply_chat_template([{"role": "user", "content": PLACEHOLDER}], tokenize=False, add_generation_prompt=True,
+                                   enable_thinking=True)
+    pre, post = chat.split(PLACEHOLDER)
+    raw = ("{%- for m in messages -%}{%- if m['content'] is string -%}{{ m['content'] }}{%- else -%}"
+           "{%- for c in m['content'] -%}{%- if c['type'] == 'text' -%}{{ c['text'] }}{%- else -%}" + prof["image"] +
+           "{%- endif -%}{%- endfor -%}{%- endif -%}{%- endfor -%}")
+    names = args.served_model_name
+    S.update(head=head, temps=temps, labels=[l for l, _ in dec], label_ids=[t for _, t in dec], ans_labels=ans, prof=prof,
+             think_pre=pre, think_post=post, raw=raw, end_think_id=tok.convert_tokens_to_ids(prof["end_think"]),
+             model=(names[0] if isinstance(names, list) else names) or args.model, model_type=mtype)
+
+
+# ------------------------------------------------------------------ helpers
+def _parts(state):
+    if isinstance(state, str):
+        return [{"type": "text", "text": state}], False
+    if isinstance(state, dict):
+        return [{"type": "text", "text": json.dumps(state, ensure_ascii=False)}], False
+    out, img = [], False
+    for p in state:
+        if isinstance(p, str):
+            out.append({"type": "text", "text": p})
+        elif isinstance(p, dict) and "image" in p:
+            out.append({"type": "image_url", "image_url": {"url": p["image"]}}); img = True
+        elif isinstance(p, dict) and p.get("type") == "image_url":
+            out.append(p); img = True
+        elif isinstance(p, dict) and p.get("type") == "text":
+            out.append(p)
+        else:
+            out.append({"type": "text", "text": json.dumps(p, ensure_ascii=False)})
+    return out, img
+
+
+def _softmax(z):
+    m = max(z); e = [math.exp(x - m) for x in z]; s = sum(e)
+    return [x / s for x in e]
+
+
+def _err(msg, code=400):
+    return JSONResponse({"error": {"message": msg, "type": "BadRequestError", "code": code}}, status_code=code)
+
+
+class Upstream(Exception):
+    def __init__(self, resp):
+        self.resp = resp
+
+
+def _check(out):
+    if hasattr(out, "error"):
+        raise Upstream(out)
+    return out
+
+
+class Ctx:
+    def __init__(self, raw):
+        self.raw, self.requests, self.prompt_tokens, self.completion_tokens = raw, 0, 0, 0
+
+    def count(self, out):
+        self.requests += 1
+        if getattr(out, "usage", None):
+            self.prompt_tokens += out.usage.prompt_tokens or 0
+            self.completion_tokens += out.usage.completion_tokens or 0
+
+
+async def _logprobs(ctx, content, has_img, ids, model, allowed=None):
+    """One-token read-out: {token_id: logprob} for `ids` after the content (text or chat parts)."""
+    lp = {}
+    for i in range(0, len(ids), CHUNK):
+        chunk = ids[i: i + CHUNK]
+        kw = dict(logprob_token_ids=chunk) if allowed is None else dict(allowed_token_ids=allowed)
+        if has_img:
+            r = ChatCompletionRequest(model=model, messages=[{"role": "user", "content": content}], chat_template=S["raw"],
+                                      add_generation_prompt=False, logprobs=True, top_logprobs=len(chunk) if allowed else 1, **kw, **READ)
+            out = _check(await ctx.raw.app.state.openai_serving_chat.create_chat_completion(r, ctx.raw)); ctx.count(out)
+            lp.update({int(t.token.split(":")[1]): t.logprob for t in out.choices[0].logprobs.content[0].top_logprobs})
+        else:
+            text = "".join(c["text"] for c in content)
+            r = CompletionRequest(model=model, prompt=text, logprobs=len(chunk) if allowed else 1, **kw, **READ)
+            out = _check(await ctx.raw.app.state.openai_serving_completion.create_completion(r, ctx.raw)); ctx.count(out)
+            lp.update({int(k.split(":")[1]): v for k, v in out.choices[0].logprobs.top_logprobs[0].items()})
+        if allowed is not None:
+            break
+    return lp
+
+
+# ------------------------------------------------------------------ System 1
+async def s1_pass(ctx, kind, parts, has_img, question, opts):
+    """One System 1 pass over `opts` (<=256). Returns probabilities aligned with opts."""
+    head, temps, prof = S["head"], S["temps"], S["prof"]
+    lo, hi = head["slots"]["ranges"][kind]
+    if kind == "choice":
+        n = len(opts); ids = S["label_ids"][:n]
+        bias = [head["bias"][lo + i] if lo + i < hi else 0.0 for i in range(n)]
+        lines = [f"{S['labels'][i]}) {o}" for i, o in enumerate(opts)]
+    else:
+        ids = head["verbalizer_ids"][lo:hi]; bias = head["bias"][lo:hi]; lines = opts
+    content = ([{"type": "text", "text": f"{prof['prefix']}[kind] {kind}\n[state] "}] + parts +
+               [{"type": "text", "text": f"\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"}])
+    lp = await _logprobs(ctx, content, has_img, ids, LORA, allowed=ids if len(ids) <= 16 else None)
+    return _softmax([(max(lp.get(t, -1e9), -1e9) + b) / temps[kind] for t, b in zip(ids, bias)])
+
+
+def _groups(n, k=16):
+    g = math.ceil(n / k); base, extra = divmod(n, g); out, i = [], 0
+    for j in range(g):
+        size = base + (1 if j < extra else 0); out.append(list(range(i, i + size))); i += size
+    return out
+
+
+async def s1_dist(ctx, kind, parts, has_img, question, opts, strategy):
+    if kind != "choice" or len(opts) <= 16 or strategy == "single":
+        return await s1_pass(ctx, kind, parts, has_img, question, opts)
+    n = len(opts)
+    if strategy == "permute":
+        import random
+        rng = random.Random(zlib.crc32(question.encode()))
+        orders = [list(range(n))] + [rng.sample(range(n), n) for _ in range(3)]
+        res = await asyncio.gather(*(s1_pass(ctx, kind, parts, has_img, question, [opts[i] for i in o]) for o in orders))
+        p = [0.0] * n
+        for o, r in zip(orders, res):
+            for i, v in zip(o, r):
+                p[i] += v / len(orders)
+        return p
+    groups = _groups(n)  # tournament: groups of <=16 in the given order (in parallel), then a final of 16
+    parts_g = await asyncio.gather(*(s1_pass(ctx, kind, parts, has_img, question, [opts[i] for i in g]) for g in groups))
+    in_group = {o: p for g, ps in zip(groups, parts_g) for o, p in zip(g, ps)}
+    chosen = [max(g, key=lambda o: (in_group[o], -o)) for g in groups]
+    rest = sorted((o for g in groups for o in g if o not in set(chosen)), key=lambda o: (-in_group[o], o))
+    fin = sorted(chosen + rest[: max(0, 16 - len(chosen))])
+    final = dict(zip(fin, await s1_pass(ctx, kind, parts, has_img, question, [opts[i] for i in fin])))
+    group_of = {o: gi for gi, g in enumerate(groups) for o in g}
+    share = [0.0] * len(groups); cap = [0.0] * len(groups)
+    for f in fin:
+        share[group_of[f]] += final[f]; cap[group_of[f]] += in_group[f]
+    among = sum(a * b for a, b in zip(share, cap))
+    p = [final[o] * among if o in final else share[group_of[o]] * in_group[o] for o in range(n)]
+    s = sum(p)
+    return [x / s for x in p]
+
+
+# ------------------------------------------------------------------ System 2 (thinking)
+async def s2_dist(ctx, kind, parts, has_img, question, opts, budget, want_text):
+    """The base model thinks over the same input; the answer-letter distribution is read after the thinking channel."""
+    shown = ["Yes (true)", "No (false)"] if kind == "noul" else opts
+    labs = S["ans_labels"][: len(shown)]
+    body = "\n".join(f"({l}) {o}" for (l, _), o in zip(labs, shown))
+    tail = (f"\n\nQuestion: {question}\n\nOptions:\n{body}\n\n"
+            "Think it through carefully, then give your final answer on the last line in the form: Answer: (X)")
+    user = [{"type": "text", "text": S["think_pre"]}] + parts + [{"type": "text", "text": tail + S["think_post"]}]
+    seed = zlib.crc32((question + body).encode()) & 0x7FFFFFFF
+    t0 = time.time()
+    gen = ChatCompletionRequest(model=S["model"], messages=[{"role": "user", "content": user}], chat_template=S["raw"],
+                                add_generation_prompt=False, add_special_tokens=False, max_tokens=budget, seed=seed,
+                                stop_token_ids=[S["end_think_id"]], skip_special_tokens=False)
+    out = _check(await ctx.raw.app.state.openai_serving_chat.create_chat_completion(gen, ctx.raw)); ctx.count(out)
+    thought = out.choices[0].message.content or ""
+    finished = out.choices[0].finish_reason == "stop"
+    ntok = out.usage.completion_tokens if out.usage else None
+    read = user + [{"type": "text", "text": thought + S["prof"]["end_think"] + S["prof"]["after_think"] + "Answer: ("}]
+    ids = [t for _, t in labs]
+    lp = await _logprobs(ctx, read, True, ids, S["model"], allowed=ids if len(ids) <= 16 else None)
+    p = _softmax([max(lp.get(t, -1e9), -1e9) for t in ids])
+    if kind == "noul":
+        p = [p[1], p[0]]  # (A) yes / (B) no -> [P(false), P(true)]
+    info = {"think_tokens": ntok, "think_seconds": round(time.time() - t0, 3), "finished_within_budget": finished}
+    if want_text:
+        info["reasoning"] = thought.replace("<|channel>thought\n", "").strip()
+    return p, info
+
+
+def _fold(p1, p2, w):
+    return [(1 - w) * x + w * y for x, y in zip(p1, p2)]
+
+
+# ------------------------------------------------------------------ routes
+router = APIRouter()
+
+
+@router.get("/v1/decide/info")
+async def info():
+    p = S["prof"]
+    return {"protocol": PROTOCOL, "model": S["model"], "model_type": S["model_type"], "max_options": len(S["labels"]),
+            "native_choice_options": len(S["head"]["slots"]["verbalizers"]) - 8, "temperatures": S["temps"],
+            "defaults": {"strategy": p["strategy"], "thinking": "off", "threshold": p["threshold"],
+                         "mix": p["mix"], "think_budget": 8192}}
+
+
+@router.post("/v1/decide")
+async def decide(req: DecideRequest, raw: Request):
+    t0 = time.time()
+    if req.kind == "choice":
+        opts = req.options or []
+        if not 2 <= len(opts) <= len(S["labels"]):
+            return _err(f"choice needs 2-{len(S['labels'])} options, got {len(opts)}")
+    else:
+        opts = ["false", "true"] if req.kind == "noul" else [str(i) for i in range(6)]
+    if req.thinking != "off" and req.kind == "score":
+        return _err("thinking is supported for noul and choice")
+    prof = S["prof"]
+    strategy = prof["strategy"] if req.strategy == "auto" else req.strategy
+    tau = prof["threshold"] if req.threshold is None else req.threshold
+    parts, has_img = _parts(req.state)
+    ctx = Ctx(raw)
+    try:
+        p1 = await s1_dist(ctx, req.kind, parts, has_img, req.question, opts, strategy)
+        probs, think = p1, None
+        if req.thinking == "on" or (req.thinking == "auto" and max(p1) < tau):
+            p2, think = await s2_dist(ctx, req.kind, parts, has_img, req.question, opts, req.think_budget, req.return_reasoning)
+            probs = _fold(p1, p2, prof["mix"])
+            think = {"used": True, **think}
+            if req.debug:
+                think.update(system1=p1, system2=p2)
+        elif req.thinking != "off":
+            think = {"used": False}
+            if req.debug:
+                think["system1"] = p1
+    except Upstream as e:
+        return JSONResponse(e.resp.model_dump(), status_code=e.resp.error.code)
+    k = max(range(len(probs)), key=probs.__getitem__)
+    native = req.kind != "choice" or len(opts) <= 16
+    resp = {"kind": req.kind, "effective_kind": req.kind, "options": opts, "probabilities": probs, "choice_index": k,
+            "choice": opts[k], "adaptation": "native" if native else f"{strategy}", "protocol": PROTOCOL, "model": S["model"],
+            "usage": {"prompt_tokens": ctx.prompt_tokens, "completion_tokens": ctx.completion_tokens,
+                      "total_tokens": ctx.prompt_tokens + ctx.completion_tokens},
+            "elapsed_seconds": time.time() - t0, "num_model_requests": ctx.requests}
+    if think is not None:
+        resp["thinking"] = {"mode": req.thinking, "threshold": tau, **think}
+    return resp
+
+
+_build_app = entry.build_app
+
+
+def build_app(args, *a, **kw):
+    app = _build_app(args, *a, **kw)
+    setup(args)
+    app.include_router(router)
+    return app
+
+
+entry.build_app = build_app
+
+if __name__ == "__main__":
+    entry.main()

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f
+size 32169626

tokenizer_config.json

@@ -0,0 +1,120 @@
+{
+  "audio_token": "<|audio|>",
+  "backend": "tokenizers",
+  "boa_token": "<|audio>",
+  "boi_token": "<|image>",
+  "bos_token": "<bos>",
+  "eoa_token": "<audio|>",
+  "eoc_token": "<channel|>",
+  "eoi_token": "<image|>",
+  "eos_token": "<eos>",
+  "eot_token": "<turn|>",
+  "escape_token": "<|\"|>",
+  "etc_token": "<tool_call|>",
+  "etd_token": "<tool|>",
+  "etr_token": "<tool_response|>",
+  "extra_special_tokens": [
+    "<|video|>"
+  ],
+  "image_token": "<|image|>",
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Gemma4Processor",
+  "response_schema": {
+    "type": "object",
+    "properties": {
+      "role": {
+        "const": "assistant"
+      },
+      "thinking": {
+        "type": "string"
+      },
+      "content": {
+        "type": "string"
+      },
+      "tool_calls": {
+        "x-regex-iterator": "<\\|tool_call>(.*?)<tool_call\\|>",
+        "type": "array",
+        "items": {
+          "type": "object",
+          "properties": {
+            "type": {
+              "const": "function"
+            },
+            "function": {
+              "type": "object",
+              "x-regex": "call\\:(?P<name>\\w+)(?P<arguments>\\{.*\\})",
+              "properties": {
+                "name": {
+                  "type": "string"
+                },
+                "arguments": {
+                  "type": "object",
+                  "x-parser": "gemma4-tool-call",
+                  "additionalProperties": {}
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "x-regex": "(\\<\\|channel\\>thought\\n(?P<thinking>.*?)\\<channel\\|\\>)?(?P<tool_calls>\\<\\|tool_call\\>.*\\<tool_call\\|\\>)?(?P<content>(?:(?!\\<turn\\|\\>)(?!\\<\\|tool_response\\>).)+)?(?:\\<turn\\|\\>|\\<\\|tool_response\\>)?"
+  },
+  "response_template": {
+    "defaults": {
+      "role": "assistant"
+    },
+    "fields": {
+      "content": {
+        "close": [
+          "<turn|>",
+          "<|tool_response>",
+          "<eos>"
+        ],
+        "content": "text"
+      },
+      "thinking": {
+        "close": "<channel|>",
+        "content": "text",
+        "open": "<|channel>thought\n"
+      },
+      "tool_calls": {
+        "close": "<tool_call|>",
+        "content": "json",
+        "content_args": {
+          "string_delims": [
+            [
+              "<|\"|>",
+              "<|\"|>"
+            ]
+          ],
+          "unquoted_keys": true
+        },
+        "open_pattern": "<\\|tool_call>call:(?P<name>\\w+)",
+        "repeats": true,
+        "transform": {
+          "function": {
+            "arguments": "{content}",
+            "name": "{name}"
+          },
+          "type": "function"
+        }
+      }
+    },
+    "start_anchor": [
+      "<|turn>model\n",
+      "<tool_response|>"
+    ]
+  },
+  "soc_token": "<|channel>",
+  "sot_token": "<|turn>",
+  "stc_token": "<|tool_call>",
+  "std_token": "<|tool>",
+  "str_token": "<|tool_response>",
+  "think_token": "<|think|>",
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>"
+}