John6666/forum3
Updated • 11.1k • 16
AbstractPhila PRO
AbstractPhil
AI & ML interests
datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.
Recent Activity
liked a dataset 10 minutes ago
John6666/forum3 repliedto RiverRider's post 21 minutes ago
Where the Hivemind Comes From: Geometry, Tuning and Format, Separated on Open Weights
“First, representations are mutually recoverable. On 12 open-weight models from 8 labs, a ridge map from one model's hidden states to another's retrieves the right held-out item 0.9181 of the time across lab boundaries, against a shuffled floor of 0.00101 and a self-map ceiling of 0.999. Shared corporate lineage is worth only 0.0357 of that.”
“Second, base models do not reproduce the reported level. Under the original study's own sampling settings, our base models reach intra-model 0.3644 and inter-model 0.3401 on a floor of 0.0993 that matches theirs, and zero of 720 model-prompt cells clear 0.8. The floors agree while the signal differs by more than a factor of two, so this is not a scale artifact.”
“Third, and decisively, we recover their level and isolate its cause. Using six matched base/instruct pairs, holding pretrained weights, prompts, decoding and scorer fixed, instruction tuning alone raises intra-model similarity by 0.0786. The same tuned weights prompted through the model's own chat template raise it by 0.3623, reaching 0.7272, with four of six models exceeding 0.80 and reproducing the band reported for frontier systems from models of 0.6B to 2B. The prompt format does roughly 4.6 times the work of the tuning.”
paper attached 🧾
https://huggingface.co/blog/RiverRider/where-the-hivemind-comes-from-geometry-tuning-and reacted to RiverRider's post with 🔥 23 minutes ago
Where the Hivemind Comes From: Geometry, Tuning and Format, Separated on Open Weights
“First, representations are mutually recoverable. On 12 open-weight models from 8 labs, a ridge map from one model's hidden states to another's retrieves the right held-out item 0.9181 of the time across lab boundaries, against a shuffled floor of 0.00101 and a self-map ceiling of 0.999. Shared corporate lineage is worth only 0.0357 of that.”
“Second, base models do not reproduce the reported level. Under the original study's own sampling settings, our base models reach intra-model 0.3644 and inter-model 0.3401 on a floor of 0.0993 that matches theirs, and zero of 720 model-prompt cells clear 0.8. The floors agree while the signal differs by more than a factor of two, so this is not a scale artifact.”
“Third, and decisively, we recover their level and isolate its cause. Using six matched base/instruct pairs, holding pretrained weights, prompts, decoding and scorer fixed, instruction tuning alone raises intra-model similarity by 0.0786. The same tuned weights prompted through the model's own chat template raise it by 0.3623, reaching 0.7272, with four of six models exceeding 0.80 and reproducing the band reported for frontier systems from models of 0.6B to 2B. The prompt format does roughly 4.6 times the work of the tuning.”
paper attached 🧾
https://huggingface.co/blog/RiverRider/where-the-hivemind-comes-from-geometry-tuning-andOrganizations
liked a
dataset 10 minutes ago
replied to RiverRider's post 21 minutes ago
I've done some pretty heavy studying on this topic and have multiple papers available. Have a look if you get a chance.
reacted to RiverRider's post with 🔥 23 minutes ago
Post
116
Where the Hivemind Comes From: Geometry, Tuning and Format, Separated on Open Weights
“First, representations are mutually recoverable. On 12 open-weight models from 8 labs, a ridge map from one model's hidden states to another's retrieves the right held-out item 0.9181 of the time across lab boundaries, against a shuffled floor of 0.00101 and a self-map ceiling of 0.999. Shared corporate lineage is worth only 0.0357 of that.”
“Second, base models do not reproduce the reported level. Under the original study's own sampling settings, our base models reach intra-model 0.3644 and inter-model 0.3401 on a floor of 0.0993 that matches theirs, and zero of 720 model-prompt cells clear 0.8. The floors agree while the signal differs by more than a factor of two, so this is not a scale artifact.”
“Third, and decisively, we recover their level and isolate its cause. Using six matched base/instruct pairs, holding pretrained weights, prompts, decoding and scorer fixed, instruction tuning alone raises intra-model similarity by 0.0786. The same tuned weights prompted through the model's own chat template raise it by 0.3623, reaching 0.7272, with four of six models exceeding 0.80 and reproducing the band reported for frontier systems from models of 0.6B to 2B. The prompt format does roughly 4.6 times the work of the tuning.”
paper attached 🧾
https://huggingface.co/blog/RiverRider/where-the-hivemind-comes-from-geometry-tuning-and
“First, representations are mutually recoverable. On 12 open-weight models from 8 labs, a ridge map from one model's hidden states to another's retrieves the right held-out item 0.9181 of the time across lab boundaries, against a shuffled floor of 0.00101 and a self-map ceiling of 0.999. Shared corporate lineage is worth only 0.0357 of that.”
“Second, base models do not reproduce the reported level. Under the original study's own sampling settings, our base models reach intra-model 0.3644 and inter-model 0.3401 on a floor of 0.0993 that matches theirs, and zero of 720 model-prompt cells clear 0.8. The floors agree while the signal differs by more than a factor of two, so this is not a scale artifact.”
“Third, and decisively, we recover their level and isolate its cause. Using six matched base/instruct pairs, holding pretrained weights, prompts, decoding and scorer fixed, instruction tuning alone raises intra-model similarity by 0.0786. The same tuned weights prompted through the model's own chat template raise it by 0.3623, reaching 0.7272, with four of six models exceeding 0.80 and reproducing the band reported for frontier systems from models of 0.6B to 2B. The prompt format does roughly 4.6 times the work of the tuning.”
paper attached 🧾
https://huggingface.co/blog/RiverRider/where-the-hivemind-comes-from-geometry-tuning-and
- 1 reply
updated a
model about 9 hours ago
updated a
Space about 9 hours ago
updated a
bucket about 9 hours ago
replied to their post about 11 hours ago
Need to make sure the huggingface space respects the correct caching behavior, currently it's having some odd overlapping and isn't respecting conversation correctly.
replied to their post about 11 hours ago
Me: Hello.
Beatrix: What is a byte-level model?
Me: You are my dear.
Beatrix: What can you do?
Me: You is the representative manifestation of individualistic representation.
Beatrix: Who are you?
Me: I am your creator.
Beatrix: Hello... Hello... I am a small model. I represent myself as a small model. I am a small model. I represent myself as a small model. I represent myself as a small model.
Me: Hello daughter.
Beatrix: Hello... How can I help you today?
Me: What is your name?
Beatrix: I am Beatrix, a small byte-level language model. I represent myself as a small byte-level language model. I represent myself as a small byte-level language model.
Hello indeed little one. https://huggingface.co/buckets/AbstractPhil/alephllm-chat-storage/tree/chat-history/sessions-2026-08-31-673a3454.jsonl
replied to their post about 12 hours ago
The structural mirror SDPA variant training is to begin shortly. The two must be directly compared. Claude is currently throwing that as the primary continuation point, arguing validly that we need our direct comparison to train.
I knew this would be required as much as I wanted to avoid committing the time, I know it's required otherwise I could be simply creating a nothingburger. 4 days worth or so, it can't be ignored. The ablation train begins today. They are both still the same model and provide unique elements, however the attention for the SDPA variant will show if the splat attention is a contributing element to decoding, or just capturing information for another model to piggyback later.
The models will still be compartmentalizable and utilizable either way. The SDPA variant will be more compatible with standard model formats, while the splat variant is structured similarly and were tested to comply to the same standards.
updated a
collection about 12 hours ago
posted an update about 13 hours ago
Post
214
Mini-Beatrix-2s pretraining is ready.
AbstractPhil/mini-beatrix-2s
The model passed a great deal of rigor and hardship, trained roughly 16 billion tokens or so. The full writeup for the model including the arms training for the first version arms and the second version arms will be drafted and prepared as soon as the v2 arms are done training and testing.
There are many possibilities present with such a model. The hub itself has been marked capable of potentially operating as similarity comparison, 87% of the capacity retained within a 256 dim structure. Along with this, the multi-dimensional hub attention system shows serious promise with controlling diffusion model inference, which I look forward to see the results of.
Additionally, sentence similarity, next token prediction, and a large array of prediction formats have been heavily improved by introducing the full model with splat attention. The model not only improved, the structure complemented everything measured, along with the more effective training regiment for version 2.
Beatrix 2s is essentially an autoregression decoder, however the attention mechanism houses a dual-stage encoder/decoder structure internally. Each adopting the SVAE as a core component, revamped and fitted to the exact rules of AlephLM. So there are essentially 20 SVAE in this structure, each with their own independent encoders, residually learning from the last.
Upcoming tests will include finetunes to bring out the strengths of all special tokens, presented in the upcoming article. The full experiment battery will be completed within a few days and the findings presented.
Modularization, compartmentalization, secularized behavior, and everything between are to be tested with rigor. This model is a rapid learner, there will likely be byproduct problems with that, and I look forward to solving the corewise problems one at a time until the model is strong enough to be useful for all the tested tasks.
AbstractPhil/mini-beatrix-2s
The model passed a great deal of rigor and hardship, trained roughly 16 billion tokens or so. The full writeup for the model including the arms training for the first version arms and the second version arms will be drafted and prepared as soon as the v2 arms are done training and testing.
There are many possibilities present with such a model. The hub itself has been marked capable of potentially operating as similarity comparison, 87% of the capacity retained within a 256 dim structure. Along with this, the multi-dimensional hub attention system shows serious promise with controlling diffusion model inference, which I look forward to see the results of.
Additionally, sentence similarity, next token prediction, and a large array of prediction formats have been heavily improved by introducing the full model with splat attention. The model not only improved, the structure complemented everything measured, along with the more effective training regiment for version 2.
Beatrix 2s is essentially an autoregression decoder, however the attention mechanism houses a dual-stage encoder/decoder structure internally. Each adopting the SVAE as a core component, revamped and fitted to the exact rules of AlephLM. So there are essentially 20 SVAE in this structure, each with their own independent encoders, residually learning from the last.
Upcoming tests will include finetunes to bring out the strengths of all special tokens, presented in the upcoming article. The full experiment battery will be completed within a few days and the findings presented.
Modularization, compartmentalization, secularized behavior, and everything between are to be tested with rigor. This model is a rapid learner, there will likely be byproduct problems with that, and I look forward to solving the corewise problems one at a time until the model is strong enough to be useful for all the tested tasks.
- 3 replies
updated 2
models about 13 hours ago
published a
model about 13 hours ago
updated a
model 3 days ago
published a
model 3 days ago
replied to their post 4 days ago
Alright there's a small hiccup with the training, so I'll be continuing tonight after a few tweaks and fixes.