Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

dnhkng.github.io

Post reply on HN
Print view View original

51–60 of 137 posts

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#51

Post by Lerc »

I have had broadly the same intuitions on the use of middle layers, but haven't had much luck with the tiny models that I can run on my hardware.

There's a video on YouTube https://www.youtube.com/watch?v=pDsTcrRVNc0

about a looping layer models, after watching that I poured some thoughts off the top of my head into a comment which, of course, promptly sunk without a trace. I'll repost the gist of them here.

If you gain benefit from looping layers, at some level every layer of parameters is in front of and behind every other, the conclusion must be that the order of the layers does not need to be fixed at all.

If you cycle through the layers multiple times, are you doing so for the benefit of a particular layer on a particular problem. If so, can you skip the other layers that don't add on repetition. If you can skip (and you can know when to skip), and you can repeat (and know when to repeat)

What you would need is a mechanism which can decide which layer is needed next. Is that then not a looping single layer MOE model? Storing the layers as a wide set of selectable options rather than a deep set of unconditional layers. You would be picking what the next layer should be (or exit the loop) the threshold for exit drops each iteration so it always eventually exits. With a tunable 'how hard to think' knob to adjust the threshold.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#52

Post by hex4def6 »

I've gotta say, this writeup gives me an itchy feeling. It really does feel like poking around a synthetic brain at this point.

You could make the argument it's closer to the blocks of a CPU compared with a brain, and it's no different to copy-pasting some IP block for eg, HW JPEG decoding. But I feel like the difference here is we're 'discovering' these blocks / organs. They weren't designed, they were evolved.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#53

Post by oliver_dr »

dormento wrote: post #47
Balinares wrote: post #23

The idea that there may be a cognitive lingua franca hiding in the layers is fascinating and gives me hope for a neat idea: pluggable knowledge banks. MoE notwithstanding, a model trained on the whole Internet and a few hundred thousands stolen books carries way more knowledge than is actually needed for any given workflow. It would be great if we could ship slimmed down models into which we'd plug the knowledge bank…

This is interesting. Would this mean less space for hallucination as well (depending on the breadth of knowledge applied to a specific task)?

[dead]

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#55

Post by phn »

A fascinating thing for me after reading this is: how can it be that the "circuit input" is compatible with its output to the point where the performance improves? The training process never saw this particular connection just like it didn't see layer 60 output into layer 3 or whatever.

Great read, makes you wonder what else is encoded in these models that might be useful!

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#56

Post by [deleted] »

Lerc wrote: post #51

I have had broadly the same intuitions on the use of middle layers, but haven't had much luck with the tiny models that I can run on my hardware. There's a video on YouTube https://www.youtube.com/watch?v=pDsTcrRVNc0 about a looping layer models, after watching that I poured some thoughts off the top of my head into a comment which, of course, promptly sunk without a trace. I'll repost the gist of them here. If you g…

[deleted]

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#57

Post by pennomi »

Balinares wrote: post #23

The idea that there may be a cognitive lingua franca hiding in the layers is fascinating and gives me hope for a neat idea: pluggable knowledge banks. MoE notwithstanding, a model trained on the whole Internet and a few hundred thousands stolen books carries way more knowledge than is actually needed for any given workflow. It would be great if we could ship slimmed down models into which we'd plug the knowledge bank…

Agreed, I suspect that LLMs in the future will have separate (possibly standardized) decoding/encoding layers that plug into logic layers.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#58

Post by idiotsecant »

dormento wrote: post #48
dnhkng wrote: post #46

Earlier quoted context omitted.

Because its generally expected that models only work 'in distribution', i.e. they work on stuff they have previously seen. They almost certainly have never seen regular conversations in Base64 in their training set, so its weird that it 'just works'. Does that make sense?

For all we know, AI tech companies could theoretically have converted all of the "acquired" (ahem!) training set material into base64 and used it for training as well, just like you would encode say japanese romaji or hebrew written in the english alphabet.

'Yes, I know we already trained on all that data, but now I want you to convert to base64 and train it again! at enormous cost!'

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#59

Post by broDogNRG »

dnhkng wrote: post #46
dinobones wrote: post #41

Earlier quoted context omitted.

why tho? it's just an alternate alphabet/set of symbols.

Because its generally expected that models only work 'in distribution', i.e. they work on stuff they have previously seen. They almost certainly have never seen regular conversations in Base64 in their training set, so its weird that it 'just works'. Does that make sense?

[dead]

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#60

Post by iamjackg »

I find the concept of LLM "brain surgery" fascinating, precisely because of how opaque the network is. One of the first things I did back when llama.cpp first got vision model support was hack the code to zero out (or otherwise modify) random numbers in the image embedding generated by the projector and then ask the LLM to describe the image. It was absolutely fascinating.

It would go from a normal description of the item in the picture to suddenly seeing people clapping in the background that were not there, or making up some other stuff. I kinda stopped after a while, but I should pick that back up and do a more coherent experiment to see if I can find any correlation between vector dimensions and "meaning."

Post reply on HN