🤝 Open to Collab

AbstractPhila PRO

AbstractPhil

19 6 30

https://civitai.com/user/AbstractPhila

AbstractEyes

AI & ML interests

datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.

Recent Activity

updated a model about 2 hours ago

AbstractPhil/alephllm-nano-beatrix-training

repliedto their post about 2 hours ago

Been a hectic week since the completion of the mini-beatrix-3 as I tested new formats for the model's v4 state. Two new model formats now exist; Castor and Pollux. They will be trained with the same data as the beatrix-2s model was given, around 30b tokens. The V4 variant of the AlephLLM dubbed Beatrix is a pair of twins particularly designed with a series of context window curated frame tokens. Castor, and Pollux. Each housing a curated frame-spaced context window with curated depth embedded frame curations. For example; a divided 4096 context window can be 1024, 512, 256, 256, 64, 64, and the rest padding. Nano-Castor-V1, the rotary softmax attention variant with around 8m params. Structurally capable of better curated "needle in a haystack" behavior. Stronger at depth and houses a cheaper format of attention than Pollux. Faster, stable by design, and only 6 blocks. Trains at about 900k tok/s. Pollux, the rotary splat/softmax hybrid trained with a format of Canon LLM inspired by music theory lookup and multiple attention hubs at around 9m params. This model houses a deeper understanding and a more complex relational behavior than simple frame lookup, but the formula is a little more complex so it takes a little longer per step. Trains at around 450k tok/s. All attributions will be given to the teams for the released papers utilized to create the components. The experiment is based on training the four models to be capable of deciding and deducting high complexity predictions of frames and the context of those frames within their own frame deduction, creating a pseudo-language. The ultimate goal to make this context window look like gibberish with a full lookup table for debugging, and out comes the correct next unicode byte. Will it work? Maybe not at this size. The tests show the frames work, the structure lines up, and the system does conform. Even if it doesn't work, the four prototypes will be ready by tomorrow evening and with those results new data

posted an update about 3 hours ago

View all activity

Organizations

Been a hectic week since the completion of the mini-beatrix-3 as I tested new formats for the model's v4 state. Two new model formats now exist; Castor and Pollux. They will be trained with the same data as the beatrix-2s model was given, around 30b tokens.

The V4 variant of the AlephLLM dubbed Beatrix is a pair of twins particularly designed with a series of context window curated frame tokens.

Castor, and Pollux. Each housing a curated frame-spaced context window with curated depth embedded frame curations. For example; a divided 4096 context window can be 1024, 512, 256, 256, 64, 64, and the rest padding.

Nano-Castor-V1, the rotary softmax attention variant with around 8m params. Structurally capable of better curated "needle in a haystack" behavior. Stronger at depth and houses a cheaper format of attention than Pollux. Faster, stable by design, and only 6 blocks. Trains at about 900k tok/s.

Pollux, the rotary splat/softmax hybrid trained with a format of Canon LLM inspired by music theory lookup and multiple attention hubs at around 9m params. This model houses a deeper understanding and a more complex relational behavior than simple frame lookup, but the formula is a little more complex so it takes a little longer per step. Trains at around 450k tok/s.

All attributions will be given to the teams for the released papers utilized to create the components.

The experiment is based on training the four models to be capable of deciding and deducting high complexity predictions of frames and the context of those frames within their own frame deduction, creating a pseudo-language. The ultimate goal to make this context window look like gibberish with a full lookup table for debugging, and out comes the correct next unicode byte.

Will it work? Maybe not at this size. The tests show the frames work, the structure lines up, and the system does conform. Even if it doesn't work, the four prototypes will be ready by tomorrow evening and with those results new data