Post
7383
stuntd 0.1.3 is out, and most of it started with a comment under my last post :)
@dipankarsarkar ran the support demo himself and showed my "1,000 tickets the heads never saw" were new wording of known tickets, not new kinds. On templates left out of training the heads were sure on 40% of tickets and right on only 61% of those. They also answered "what's the weather in Paris" like it was a billing ticket.
So now every head keeps the encoder vectors of what it trained on, and anything far from all of them goes to the big model no matter how confident the head is.
Screenshot is the same three requests with the gate off and on. It's local Jev mode with no provider, so the gate hands them to zero-shot Laya. Behind OpenAI or Anthropic they'd go to your model.
On the support demo it costs about 6 points of local answers on normal tickets (76.6% to 70.6%, still 97% right), and tickets from unseen templates go from 40% answered locally to 0. Funny detail: a ticket I typed by hand counted as new too, these demo heads only ever saw 20 templates.
Also in this one: auto retrain counts distinct texts and waits between runs, report shows intervals, import skips duplicates, site rm/rename, /healthz and stuntd stop.
https://github.com/bladedevoff/stuntd/releases/tag/v0.1.3
@dipankarsarkar ran the support demo himself and showed my "1,000 tickets the heads never saw" were new wording of known tickets, not new kinds. On templates left out of training the heads were sure on 40% of tickets and right on only 61% of those. They also answered "what's the weather in Paris" like it was a billing ticket.
So now every head keeps the encoder vectors of what it trained on, and anything far from all of them goes to the big model no matter how confident the head is.
Screenshot is the same three requests with the gate off and on. It's local Jev mode with no provider, so the gate hands them to zero-shot Laya. Behind OpenAI or Anthropic they'd go to your model.
On the support demo it costs about 6 points of local answers on normal tickets (76.6% to 70.6%, still 97% right), and tickets from unseen templates go from 40% answered locally to 0. Funny detail: a ticket I typed by hand counted as new too, these demo heads only ever saw 20 templates.
Also in this one: auto retrain counts distinct texts and waits between runs, report shows intervals, import skips duplicates, site rm/rename, /healthz and stuntd stop.
pip install -U stuntdhttps://github.com/bladedevoff/stuntd/releases/tag/v0.1.3
