v13: production-engine score added, hivetrace ru F1 0.59->0.80, MUST 21/21

README.md

@@ -32,46 +32,67 @@ does (see `touch_rate` below).
 
 ## Verified results
 
-Gate run 2026-08-30. Every figure measured, none estimated.
+Gate run 2026-09-03 (internal build v13), scored through the production Rust
+inference engine, not a Python decode of the same ONNX. Every figure measured,
+none estimated.
 
-| | this model | model it replaces |
+| | this model | previously shipped |
 |---|---|---|
-| **regex-owned identifiers wrongly touched** (real ru text) | **0.1%** (1/770) | 53.6% (413/770) |
-| **known-defect suite** | **18/18** | 7/18 |
-| hivetrace ru PII — F1 / P / R | 0.5888 / 0.4375 / **0.9000** | 0.2754 / 0.1610 / 0.9500 |
-| Wojood ar NER — F1 | 0.4888 | 0.4136 |
-| int8 F1 (held-out synthetic) | 0.8906 | 0.8704 |
-| int8 vs fp32 F1 cost | 0.0081 | — |
-| latency p95 @512 tok (CPU, 4 thread) | 190.9 ms | 190.8 ms |
-| name-origin bias delta | 0.0312 | not validly measured |
-
-Head-to-head vs the previously shipped multilingual PII model, ours ahead in
-every language: it +0.58 · es +0.55 · fr +0.52 · en +0.52 · de +0.48 ·
-ar +0.45 · **ru +0.07**.
+| **regex-owned identifiers wrongly touched** (real ru text) | **0.0%** (0/770) | 0.1% (1/770) |
+| **known-defect suite** | **21/21** | 18/18 |
+| hivetrace ru PII — F1 / P / R | 0.8036 / 0.6888 / **0.9643** | 0.5888 / 0.4375 / 0.9000 |
+| Wojood ar NER — F1 / P / R | 0.4945 / 0.6143 / 0.4137 | 0.4888 / — / — |
+| production-engine filtered F1 (held-out synthetic) | **0.8563** | 0.7357 |
+| int8 F1 (held-out synthetic, Python decode) | 0.9321 | 0.8906 |
+| int8 vs fp32 F1 cost | 0.0025 | 0.0081 |
+| latency p95 @512 tok (CPU) | 193.4 ms | 190.9 ms |
+| name-origin bias delta | 0.0469 | 0.0312 |
+
+Head-to-head vs the incumbent it replaces (`onnx-community/multilang-pii-ner-ONNX`,
+PERSON only, WikiANN), ours ahead in every language: es +0.56 · it +0.53 ·
+en +0.55 · fr +0.51 · de +0.51 · ar +0.51 · ru +0.12 (95% CI entirely above the
+`-0.02` no-regression floor for every language).
+
+**The production-engine number is new in this run and is the one that matters.**
+Earlier cards quoted only a Python decode of the ONNX graph — the actual Rust
+inference engine (chunking, tokenizer template, span filtering) had never been
+scored directly, and doing so once already surfaced and fixed a real skew (the
+engine was dropping the `<s>`/`</s>` special tokens the model was trained with).
+0.8563 is what the shipped agent actually emits on held-out synthetic text.
+
+**Latency and bias moved slightly against this model, both still inside
+budget.** p95 latency rose 2.5 ms (ceiling is 200 ms) and bias delta rose 0.0157
+(budget is 0.05). Neither regression is large enough to matter on its own; noted
+for anyone tracking the trend across runs rather than a single gate pass.
 
 ## Read this before using it
 
-**Recall was traded for precision.** Recall on real Russian PII fell 0.95 → 0.90
-while precision rose 0.16 → 0.44. False positives dropped roughly 500×, which is
-the right trade for a tool that rewrites user text — but if your application
-prefers over-flagging to under-flagging, the model it replaces had higher recall.
+**Recall was traded for precision, then partly traded back.** The previously
+shipped model already cut false positives on structured identifiers roughly
+500× versus its own predecessor (53.6% → 0.1% touch rate); this run holds that
+line at 0.0% while also lifting real Russian PII recall 0.90 → 0.96 and F1
+0.5888 → 0.8036 — the biggest jump this model line has made on real text in one
+step.
 
-**Russian is the weak language.** +0.07 head-to-head where every other language
-is +0.45 or better, and hivetrace F1 0.59 is the lowest real-text score we have.
-
-**Organisation boundaries are imperfect.** On
-`вадим бельский из клиники «Здоровье»` it returns `клиники` (the common noun
-"clinic") rather than the quoted name `«Здоровье»`. It finds an organisation is
-present; it does not always choose the right span.
+**Organisation boundaries are still imperfect.** Wojood F1 is barely moved
+(0.4888 → 0.4945) and organisation spans remain the weakest category on real
+Arabic text — expect it to sometimes find that an organisation is present
+without choosing the exact right span.
 
 **Trained only on synthetic data.** Entity pools are finite, so scores on
-generated text overstate real-world ability — the same model scores 0.95 on
-held-out synthetic and 0.59 on real Russian. Judge it on the external
-benchmarks above, not the synthetic figure.
+generated text overstate real-world ability. Judge it on the external
+benchmarks above (hivetrace, Wojood), not the synthetic figure.
 
 **Arabizi is a reconstruction.** No corpus contains it; its conventions
 (3=ain, 7=haa, 2=hamza) are our model of how people type, not observed data.
 
+**DATE_TIME scores 0.000 in the production-filtered number by design, not by
+defect.** The production filter keeps a date span only next to an explicit
+birth-context cue (`born on`, `DOB:`, …); this eval corpus carries none, so
+every date the model correctly finds is filtered back out before it reaches
+the number above. It is not gated and should not be read as a date-detection
+failure — see `precision.rs`'s date-context rule in the agent's source.
+
 ## Use
 
 Three files: `config.json`, `tokenizer.json`, `onnx/model_quantized.onnx`.
@@ -93,14 +114,19 @@ logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
 # argmax -> id2label -> merge BIO spans using `offsets` for character positions
 ```
 
+**Include `<s>`/`</s>` in the tokenizer call for full-fidelity results** — this
+is what closed most of the gap between the Python-decode number and the
+production-engine number above. `AutoTokenizer.__call__` does this by default;
+only a hand-rolled encode path that skips special tokens needs to add them back.
+
 Pair it with a regex/checksum layer for the structured identifiers it
 deliberately ignores.
 
 ## Training data
 
-`ScienceSoft/scnsoft-pii-synthetic-corpus` — 486,947 rows, fully synthetic, no
-real person's data. Roughly 13% of rows contain no entity at all; those
-entity-free negatives are what took `touch_rate` from 53.6% to 0.1%.
+`ScienceSoft/scnsoft-pii-synthetic-corpus` — fully synthetic, no real person's
+data. A meaningful share of rows contain no entity at all; those entity-free
+negatives are what keeps `touch_rate` near zero.
 
 ## Licence
 

onnx/model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bc5329911105c3d4a0cb95c0c18823c6e2575866b085633decc51523327a2f3b
+oid sha256:3e7189f37871b9d261edc9f8a771651459b4e20a3d23506100d10e3ed7a10ff1
 size 278234412

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}