Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

dnhkng.github.io

Post reply on HN
Print view View original

111–120 of 137 posts

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#111

Post by adcoleman6 »

hex4def6 wrote: post #52

I've gotta say, this writeup gives me an itchy feeling. It really does feel like poking around a synthetic brain at this point. You could make the argument it's closer to the blocks of a CPU compared with a brain, and it's no different to copy-pasting some IP block for eg, HW JPEG decoding. But I feel like the difference here is we're 'discovering' these blocks / organs. They weren't designed, they were evolved.

The difference is less stark these days, with generative design being used for semiconductors.

Altering these features isn’t messing with evolution anymore than tweaking a CAD file that used genetic algorithms: it’s all math, 1s and 0s.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#112

Post by adcoleman6 »

idiotsecant wrote: post #58
dormento wrote: post #48

Earlier quoted context omitted.

For all we know, AI tech companies could theoretically have converted all of the "acquired" (ahem!) training set material into base64 and used it for training as well, just like you would encode say japanese romaji or hebrew written in the english alphabet.

'Yes, I know we already trained on all that data, but now I want you to convert to base64 and train it again ! at enormous cost!'

On the contrary, it could be a deliberate attempt to augment or diversify the dataset.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#113

Post by koenschipper »

This is an incredibly elegant hack. The finding that it only works with "circuit-sized" blocks of ~7 layers is fascinating. It really makes you wonder how much of a model's depth is just routing versus actual discrete processing units.

I spend a lot of time wrestling with smaller LLMs for strict data extraction and JSON formatting. Have you noticed if duplicating these specific middle layers boosts a particular type of capability?

For example, does the model become more obedient to system prompts/strict formatting, or is the performance bump purely in general reasoning and knowledge retrieval?

Amazing work doing this on a basement 4090 rig!

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#115

Post by patchnull »

dnhkng wrote: post #24
patchnull wrote: post #21

[flagged]

Have a look at the boundaries in the heatmaps. They are of course open to interpretation, but it suggest to me that the models develop 'organs' for processing different types of data, and without duplicating the 'whole organ' you don't get the benefits. This is quite different to what you usually see, which is via layer ablation experiments. Thoughts?

[flagged]

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#116

Post by JimSanchez »

Fascinating idea that LLM performance might improve simply by changing the inference path through existing layers rather than retraining weights. It’s interesting to think of transformer stacks developing something like functional “circuits” similar to brain regions.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#117

Post by janalsncm »

gwern wrote: post #93
janalsncm wrote: post #45

Earlier quoted context omitted.

Based on Karpathy’s writeup the auto research would not have found this. He tells the agent to improve the model and training loop with a five minute time limit, but honestly this “hack” is so far out of distribution that it seems really unlikely an agent would find this.

Adding, swapping, or duplicating layers has a long history (eg. StyleGAN, upcycling), and it was pointed out at least as far back as He et al 2015 (Resnets) that you could ablate or add more layers because they functioned more as just doing some incremental compute iteratively, and many of them were optional. (Or consider Universal Transformers or heck, just how BPTT works.) So this idea is not far out of distributio…

I don’t disagree, but it’s worth having a look at the changes the LLM did apply.

https://github.com/karpathy/autoresearch/blob/master/progres...

My opinion is you’d have to go pretty far down the x axis to get to anything that’s not things like tinkering with bs, lr, or positional encodings. There are so many hyperparameter knobs already exposed that duplicating layers is unlikely to be proposed for a long time.

I also just noticed that the last change it applied was changing the random seed. Lol.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#118

Post by gwern »

janalsncm wrote: post #117
gwern wrote: post #93

Earlier quoted context omitted.

Adding, swapping, or duplicating layers has a long history (eg. StyleGAN, upcycling), and it was pointed out at least as far back as He et al 2015 (Resnets) that you could ablate or add more layers because they functioned more as just doing some incremental compute iteratively, and many of them were optional. (Or consider Universal Transformers or heck, just how BPTT works.) So this idea is not far out of distributio…

I don’t disagree, but it’s worth having a look at the changes the LLM did apply. https://github.com/karpathy/autoresearch/blob/master/progres... My opinion is you’d have to go pretty far down the x axis to get to anything that’s not things like tinkering with bs, lr, or positional encodings. There are so many hyperparameter knobs already exposed that duplicating layers is unlikely to be proposed for a long time. I al…

My understanding was that Autoresearch was defined as training from scratch (since it's based on the nanogpt speedrun), not using any pretrained models. So it couldn't do anything like upcycling a pretrained model or the Frankenmerge, because it's not given any access to such a thing in the first place. (If it could, the speedrun would be pointless as it would mostly benchmark what is the fastest fileserver you can download a highly compressed pretrained model checkpoint from...) It can increase the number of layers for a new architecture+run, but that's not the same thing.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#119

Post by evangambit »

wolttam wrote: post #101
mysteria wrote: post #61

The astounding thing about Goliath wasn’t that is was a huge leap in performance, it was that the damn thing functioned at all. To this day, I still don’t understand why this didn’t raise more eyebrows. This wasn't something I really dug into in great detail but I remember my surprise back then at how all those merged models and those "expanded" models like Goliath still generated coherent output. IMO those were more…

I think the main challenge with combining layers of different would models be their differing embedding sizes and potentially different vocabularies. Even between two models of identical architecture, they may have landed on quite different internal representations if the training data recipe was substantially different. But it would be fun to experiment with.

Even with the same embedding sizes and vocabularies, there’s nothing that forces the meaning of dimension 1 of model 1 to mean the same thing as dimension 1 of model 2 — there are lots of ways to permute the dimensions of a model without changing its output, so whatever dimension 1 means the first time you train a model is just as likely to end up as dimension 2 the second time you train is as it is to be consistent with the first model.

Nobody here or on Reddit has mentioned this, maybe bc it’s too obvious, but it’s clear to me that the residual connections are an absolutely necessary component to making this merging possible — that’s the only reason dimension 1 of a later layer is encouraged to mean something similar to dimension 1 of an earlier layer.

Post reply on HN