Got a little ahead of myself here. We have one more phase to go before we begin. Possibly ready tonight or tomorrow morning. possibly sooner. If it's put off I'll write another post. The final layer's huge direction pressure needs to be addressed, and we have some potential fixes but they are untested. I'll need to run those first.

The idea here is to capture the amplification throughout the model rather than allowing it to become a causal obstruction in the final layer. I'd like to call it a fractal sift but it's really a lot simpler than that. Essentially giving the model the capacity to internalize the structure so the larger behavior can cut through rather than the heavier elements causing a clog at the final block.

AH it's a Muon byproduct according to the paper. That makes more sense. I'll try enabling weight decay on AdamW for the final block, and for the rest Adam no decay with Muon. I have had a strict barre on AdamW for quite a while now, as it would mask the results of problem architecture. At this point I think AdamW's weight decay is the solution to the final layer due to the methodology the optimizer employs and the byproduct the optimization has for the huge directional accumulation, so I don't need architecture I just need a couple tweaks.

Lets test that theory.

If this theory holds, I'll run tests using AdamW on the overall model rather than just on the final layer. It's possible the AdamW and the model are noncompliant though, which has been historically shown on a multitude of trains related to the aleph structures. If this is the case and they are noncompliant I will likely create a compliant version of AdamW specifically curated for the aleph structure for testing. I doubt I can create something as potent as AdamW, I just need to tweak it a little at most, but it may need to be specifically tweaked for the more errant curves that are being optimized away rather than utilized. Adam captures these a bit more effectively and I'm concerned the decay will destroy them as it did before, but I'm optimistic.

I'm approaching this one with a scalpel and a microscope.

As this may solve the faults with the aleph pred in it's entirety with a little luck. Just enabling weight decay alone may solve the problem, or it may completely brick the model. Small experiments first.

Both good news, and bad news.

The good news, I solved the final direction problem.

canary_acc 0.000 · doc n=70 bpb=4.22 reset=2.47 · bits by position (row) 0-64 1.84 64-256 1.67 256-1024 1.58 1024-2048 1.52 2048-4096 1.53
[lastfix] c-head-adam step 1,500: val 1.5488 bits a byte | share along the last block's direction 0.324 (after the norm 0.295) | last bank write 0.249 on its direction, size 328 | last block rank 35 | output layer size 12.6, content loudness 1.25


canary_acc 0.000 · doc n=70 bpb=4.17 reset=2.37 · bits by position (row) 0-64 1.77 64-256 1.60 256-1024 1.50 1024-2048 1.44 2048-4096 1.45
[lastfix] c-head-adam step 2,000: val 1.4719 bits a byte | share along the last block's direction 0.313 (after the norm 0.285) | last bank write 0.229 on its direction, size 455 | last block rank 39 | output layer size 13.1, content loudness 1.31

Probably.

THE BAD NEWS, I'll need to run all of yesterdays tests again with the new optimizer. It invalidates many of the framing experiment guidelines without running the full test suite. The models simply cannot TRAIN FOR WEEKS without running full barrage testing on the structure, in a day that could save a month of training.

It's both irresponsible and painful on the wallet if I don't run the correct tests for speed, structural architecture, and accuracy before I run.

All the fixes work. Doesn't even need weight decay, just Adam. Eliminate Muon from the final layer's head, that's all it really needs to have. The most dominant pieces the model would have directional bias formed from, solved.

Castor's early setup is clean. Very clean. I expect a strong feature by the end of training, rather than a single dominant fractal direction that would take me weeks to investigate.

As before the aleph head is getting buried. The model isn't going to be incentivized to learn nonlinear structure without a fuss. Muon made this more than viable but it formed the cascade knot that would crush the final layer. There's likely a cascade middle-ground between behavioral removal and cascade articulation. Something that will form reasonable amounts of utility without forcing the model to learn something that will simply be optimized away.

They must LEARN to cooperate without smothering each other. MUON makes this viable, but there are cascade failure points from this that cannot go unaddressed.

We have our core optimization fix. Now I need to enable the correct Cantor Aleph-Void behavior to shortcut with the aleph addressing with something similar to the applied Fractal Wormhole positions as experimented with in the past, aka matching numbers down the chain that usually lead to the classic Cantor Dust problems. Number systems that have given mathematicians headaches for many years. https://en.wikipedia.org/wiki/Cantor_set

After that the models need to learn to depend on them, which should fit nicely into the context window framing. I believe this should be stable enough under fp32 to round since I'm not using the Cantor structure for positioning directly but instead the Aleph mathematics for the actual prediction system. This should be stable enough, but if not I'll work something else out. As long as we don't hit something as complex as the Mandelbrot it shouldn't collapse or explode. FP64 isn't on the menu thank you.

A fully represented Cantor Dust byte prediction system would take a considerable amount longer per token. This is a cultivated behavioral response system meant to replicate the activity through linear mathematics, not a direct 1:1 clone.

Every fractal slice will cost huge amounts of compute, which is the entire point of avoiding using them directly and instead cultivating them as a cooperative element. Without the MUON being fully dominant they are not cultivating properly on their own, as the steps aren't allowed to form those extra potentially incorrect distances that allow nonlinear shortcuts to form. Adam and AdamW turn those into smooth curves, when we want some gaps and rifts in order to learn the nonlinear math in a way that doesn't form a billion unmeasurable infinities, or entirely matching cosine final layers.

The reason why there's a component named governor is to limit these leaps and bounds. They will eventually cause much worse problems if they are left completely unchained. So a couple hundred million per 60 billion are casualties, give or take, for those who reach too far or are pinched too tightly for rounding to correct, and are rewarded for the action.

Smooth TOO MUCH and the head never activates. Smooth TOO LITTLE and you get the big clump. Likely will need an optimizer for the behavior.

Okay so it may or may not work the same way as intended right now, however I'm not willing to abandon the semi-functional systems in favor of a formula switch that requires multiple new components. I'm not certain, and I will need to be certain.

I'll need to sleep on it. The model is far from dead, just needs a proper optimizer to cultivate the rifts. Something that handles the correct intention and the correct implicit skips as per the necessary geometric framing tested. There will need to be a tweak or two to the formula most likely, hopefully topical tweaks only. I don't want to be at the blackboard for another month only to find a model that runs like a snail on a salty day.

If the MUON system simply wasn't cultivating the results, I will figure it out tomorrow when i wake up and then figure out the correct answer.

Experiments are humming along. Ready by this evening is likely not going to be the case, however I believe I have a few solutions for the aleph system.

Historic experimentation had a few leads, mostly dead ends. A couple of options showed promise; such as treating the signal as amplitude, while other options such as the heavily vetted Cantor positioning system are sidelined for stability in this version.

The result is likely going to treat that particular Aleph signal as confidence, as removing it from the model completely caused high confidence with incorrect answers while present the confidence would dip. It very would could be correlation without causation, but the tests are worth running. I will continue to investigation to get the correct answer for the causal behavior exhibiting the result, but it's definitely been difficult to tack down.

In any case, the last block large direction problem has been solved.

The prototypes will be trained in AdamW and Muon. AdamW shows stronger frame curation and control than Adam, and the large final direction being given AdamW with weight decay. The full prototypes will likely be trained with weight decay, as the prototypes have shown strong results with it.

As long as the final block is given Adam in the right place, the model converges, and the Aleph head quiets. To cultivate this head, the aleph system will be given the job of confidence. It's not as powerful as the alternative in theory, but the pragma shows the likelihood of it working to be considerably higher and more useful.

I'll need some days to align the next nonlinear prototype head, hopefully by the end of next week. In any case, the model itself still uses aleph internally, but the global aleph system will be quiet on this version. The next byte framing and context window will be more dominant, along with the banks. This is fine as many of these prototypes can be trained.

Stability is key, as long as the model is stable the models will work. If they aren't stable or collapse then they won't be of any use for data, other than trying to prevent or balance around it.

Legitimate strengths the other does not possess. Pragma and proof with cooperation.

Unironically, the answer for the splat problem was... a form of mathematics derived from art. The Canon procedure, which a specific paper's work adapted and fitted to Pollux and the splat layer in a meaningful manner. This brought the frame system from nearly nothing on recall, to a direct comparator to a rotary softmax attention. This when applied with rotary positions allowed the splat layers to behave and frame more effectively than the tested softmax rotary scales tested on many fronts. The cost of train went up a little, but the prototype does have meanginful frame learning and context awareness.

I've predominantly considered the splat component to be an artist component in many ways. It's reasonable to find many answers to the artist's quirks for organization, in the theory of music itself. It'll be in the writeup.

https://arxiv.org/abs/2512.17351

Too much data. Needed to purge old trains, loras, and unnecessary checkpoints on the SD15, SDXL, and multiple other diffusion trains.

Mostly purged optimizers but some checkpoints went as well, mostly useless checkpoints.

Framed training going well, going to put in a selector system for it to test some CoT multi-chains and start up an embedding prototype. I'm thinking multi-thinking is a viable option for some tinkering.