Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

I think the main challenge with combining layers of different would models be their differing embedding sizes and potentially different vocabularies.

Even between two models of identical architecture, they may have landed on quite different internal representations if the training data recipe was substantially different.

But it would be fun to experiment with.

Cheers. I will go back though my other old projects (optogenetics, hacking Crispr/CAS9 etc), and put them on my blog.

On your questions: 1) A few other papers have been mentioned in the thread, like Solar10.7B. They duplicated the whole transformer stack, and it kinda helped. But as I found experimentally, that probably not a great idea. You are duplicating 'organs' (i.e. input processing stuff), that should only have one copy. Also, that paper didn't see immediate improvements; they had to do continued pre-training to see benefits. At that point, I'm guessing the big labs stopped bothering. Limited by hardware, I had to find unusual angles to approach this topic.

2) Nah, no more wetware for me. I did a half decade of research at a big neurobiology institute, and while it was very enjoyable, I can truly say that grant writing and paper review are 'not my thing'. This reason this info was delayed so long is that I wanted a paper in the AI field to go along with my papers in other fields. But as a Hobbyist with no official affiliation, and the attention span of a gnat, I gave up and started a blog instead. Maybe someone will cite it?

Yes, thats true.

But that points again to the main idea: The model has learnt to transform Base64 into a form it can already use in the 'regular' thinking structures.

The alternative is that there is an entire parallel structure just for Base64, which based on my 'chats' with LLMs in that format seems implausible; it acts like the regular model.

If there is a 'translation' organ in the model, why not a math or emotion processing organs? Thats what I set out to find, and are illustrated in the heatmaps.

Also, any writing tips from the Master blogger himself? Huge fan (squeal!)

That's really interesting. Makes me immediately ask two questions:

1. Should we be training models like this from the start? It seems that a model trained with layer loops would be able to take advantage of it better than rearranging the layers of a naive model.

2. Should we even be using a fixed number of layers? If models are this tolerant to their inner layers being meddled with, then it doesn't make sense to run all the layers on every single token.

Maybe we could make a model that changed the number of iterations through the compute layers based on how much computation it thought the problem needed. Send it through only once for easy problems (perhaps even zero times?) and two or more times for harder problems. This would allow easier prompts to complete faster, while allowing the model to potentially scale up to infinity hard problems.

If we are training or fine tuning the model, we can probably make the compute layers generate a confidence signals based that predicts how likely it is for an extra compute iteration to meaningfully change the result.

This is fascinating. The fact that only ~7 layer blocks work and not fewer/more really suggests there are emergent functional units in the transformer stack that we don't fully understand yet. Almost like "organs" in the network. Have you tried this on architectures other than Qwen, like Llama or Mistral? Curious if the magic block size is architecture-dependent or if 7 layers is some kind of universal constant.