[flagged]
Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
21–30 of 137 posts
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#22This reminds me when people were doing crazy stuff to improve the first Stable Diffusion model by swapping layers, interpolating weights, documenting which layer was most responsible for the quality of the hands etc. At the end the final models had dozens of different ancestors.
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#23The idea that there may be a cognitive lingua franca hiding in the layers is fascinating and gives me hope for a neat idea: pluggable knowledge banks.
MoE notwithstanding, a model trained on the whole Internet and a few hundred thousands stolen books carries way more knowledge than is actually needed for any given workflow. It would be great if we could ship slimmed down models into which we'd plug the knowledge banks useful for today's work, and only those.
It would also mean that you could keep a model's knowledge fresh without retraining the whole of it.
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#24[flagged]
Have a look at the boundaries in the heatmaps.
They are of course open to interpretation, but it suggest to me that the models develop 'organs' for processing different types of data, and without duplicating the 'whole organ' you don't get the benefits.
This is quite different to what you usually see, which is via layer ablation experiments. Thoughts?
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#25Isn't this similar to models that have "double check the answer"? First pass runs your input through, second pass runs it's output as input? Just, in double check it presumably runs the entire stack while you're trying to skip the translation steps and only double check the logic?
Maybe, but the interesting thing for me it this only works with specific 'chunks' of the transformer layer stack. More or less that the optimal leads to worse performance.
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#26I really enjoyed reading this. I feel like generalists intuitively experience this exact thing so much throughout their lives because they must have this neuroanatomy you describe. There’s a certain geometry to knowledge that makes possible for this orthogonal movement and it is really fascinating to me. Thank you for publishing this, you made my day!
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#27I think you may have cracked latent space reasoning. I've had a hunch that something like this would work, but couldn't figure out how the training would back propagate. But you've shown that you just need to duplicate existing layers. Have you tried a simple inline loop over the duplicated layers? Would be interesting to see performance. Also, would be interesting to compare with a MOE model. See if these layers are…
Yes, I've tried duplicating indvidual layers, but its not useful.
I think this hasn't been tried before because it's totally unintuitive that feeding the output from later layers into previous ones would actually do anything. And in fact, it usually is detrimental. I guess it takes really bored hobbyists with too much compute to check this stuff.
I have done some interesting work on applying multiple layer duplications in different regions of the model too, going so far as to train a meta-model (actually just XGBoost) to predict the merges. Seems to work, buts thats a whole other blog post.
This works with MoE, and yes, I would be interested in looking into this in more detail. But my wife might disagree with this time sink...
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#28very awesome writeup, glad to see someone with access to hw actually playing with this.
Hopefully the cost per GPU will kick-it soon and we'll see people properly play, but frankly the "middle section" layers 2(ish) to (n-1)(ish) of a model can be shuffled up/down and left/right and still perform well.
The fun one will be an LLM router for LLM layers to apply the best reasoning to the best input so far, but frankly that would need the years and years of training that the author hints at.
The one that's still out of grasps is still how to combine/manipulate per-layer k,v caches into a globally coherent state. i.e. if layers can be moved up/down why can't the cached k,v be swapped/combined with different projections? global k,v caches work, but they have to be _huge_ in order to prevent model collapse even on something as simple as owt.
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#29I really enjoyed reading this. I feel like generalists intuitively experience this exact thing so much throughout their lives because they must have this neuroanatomy you describe. There’s a certain geometry to knowledge that makes possible for this orthogonal movement and it is really fascinating to me. Thank you for publishing this, you made my day!
Thanks!
Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs
#30Crazy writeup.
Author is right about the base64 part. Does seem weird that it can decode and understand it at same time. And I guess what makes it weird that we just sorta accept that for say English and German this works ie normal use but when framed as base64 then it suddenly stops feeling intuitive