Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

dnhkng.github.io

Post reply on HN
Print view View original

91–100 of 137 posts

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#91

Post by gwern »

dnhkng wrote: post #46
dinobones wrote: post #41

Earlier quoted context omitted.

why tho? it's just an alternate alphabet/set of symbols.

Because its generally expected that models only work 'in distribution', i.e. they work on stuff they have previously seen. They almost certainly have never seen regular conversations in Base64 in their training set, so its weird that it 'just works'. Does that make sense?

> They almost certainly have never seen regular conversations in Base64 in their training set, so its weird that it 'just works'.

People use Base64 to store payloads of many arbitrary things, including web pages or screenshots, both deliberately and erroneously, and so they have almost certainly seen regular conversations in Base64 in their 10tb+ text training sets scraped from billions of web pages and files and mangled emails etc.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#93

Post by gwern »

janalsncm wrote: post #45
Aditya_Garg wrote: post #40

Wild stuff and great read Do you think karpathy's autoresearch would be useful here?

Based on Karpathy’s writeup the auto research would not have found this. He tells the agent to improve the model and training loop with a five minute time limit, but honestly this “hack” is so far out of distribution that it seems really unlikely an agent would find this.

Adding, swapping, or duplicating layers has a long history (eg. StyleGAN, upcycling), and it was pointed out at least as far back as He et al 2015 (Resnets) that you could ablate or add more layers because they functioned more as just doing some incremental compute iteratively, and many of them were optional. (Or consider Universal Transformers or heck, just how BPTT works.) So this idea is not far out of distribution, if at all, especially if you're a LLM who knows the literature and past approaches (which most humans would not because they only just got into this area post-ChatGPT).

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#94

Post by CuriouslyC »

ay wrote: post #90
Balinares wrote: post #23

The idea that there may be a cognitive lingua franca hiding in the layers is fascinating and gives me hope for a neat idea: pluggable knowledge banks. MoE notwithstanding, a model trained on the whole Internet and a few hundred thousands stolen books carries way more knowledge than is actually needed for any given workflow. It would be great if we could ship slimmed down models into which we'd plug the knowledge bank…

Isn’t that what LoRA does ?

LoRAs are better at steering models to produce correct answers from their data set than imparting new knowledge.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#97

Post by nixon_why69 »

phn wrote: post #55

A fascinating thing for me after reading this is: how can it be that the "circuit input" is compatible with its output to the point where the performance improves? The training process never saw this particular connection just like it didn't see layer 60 output into layer 3 or whatever. Great read, makes you wonder what else is encoded in these models that might be useful!

I think the intuition is that the first N layers decode into "thought language" while the last N encode back to desired output language. So if there are well defined points where it transitions between decoding/understanding, thinking, and rendering back to language, those 2 transition points should be in the same vector space of "LLM magic thinking language".

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#100

Post by twotwotwo »

This is fascinating, and makes me wonder what other things that 'should' be impossible might just be waiting for the right configuration to be tried.

For example, we take for granted the context model of LLMs is necessary, that all you can do is append and anything that changes the beginning requires a recalculation of whatever comes after it. And that does match how training works.

But all sorts of things would become possible if it were possible to shift things in and out of context without recomputing it all; conservatively you could avoid compaction, optimistically it might be a way to get info to the model that's both more deeply integrated than search and more efficient than training larger and larger models.

Post reply on HN