Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

news.mit.edu

Post reply on HN
Print view View original

41–50 of 156 posts

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#41

Post by bionhoward »

ramses0 wrote: post #22
Filligree wrote: post #8

Earlier quoted context omitted.

It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.

Is there some sort of "LLM-on-Wikipedia" competition? ie: given "just wikipedia" what's the best score people can get on however these models are evaluated. I know that all the commercial ventures have a voracious data-input set, but it seems like there's room for dictionary.llm + wikipedia.llm + linux-kernel.llm and some sort of judging / bake-off for their different performance capabilities. Or does the training tr…

Yes, that’s known as the Hutter Prize http://prize.hutter1.net/

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#42

Post by Kuinox »

Culonavirus wrote: post #37
whatever1 wrote: post #4

Llms seem like a good compression mechanism. It blows my mind that I can have a copy of llama locally on my PC and have access to virtually the entire internet

Yea except it's a lossy compression. With the lost part being hallucinated in at inference time.

If you've read the article, the LLM hallucinations aren't due to the model not knowing the information but a function that choose to remember the wrong thing.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#43

Post by learned »

i5heu wrote: post #23

So it is entirely possible to decouple the reasoning part from the information part? This is like absolutely mind blowing if this is true.

A big caveat mentioned in the article is that this experiment was done with a small set (N=47) of specific questions that they expected to have relatively simple relational answers:

> The researchers developed a method to estimate these simple functions, and then computed functions for 47 different relations, such as “capital city of a country” and “lead singer of a band.” While there could be an infinite number of possible relations, the researchers chose to study this specific subset because they are representative of the kinds of facts that can be written in this way.

About 60% of these relations were retrieved using a linear function in the model. The remaining appeared to have nonlinear retrieval and is still a subject of investigation:

> Functions retrieved the correct information more than 60 percent of the time, showing that some information in a transformer is encoded and retrieved in this way. “But not everything is linearly encoded. For some facts, even though the model knows them and will predict text that is consistent with these facts, we can’t find linear functions for them. This suggests that the model is doing something more intricate to store that information,” he says.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#44

Post by krainboltgreene »

whatever1 wrote: post #4

Llms seem like a good compression mechanism. It blows my mind that I can have a copy of llama locally on my PC and have access to virtually the entire internet

> have access to virtually the entire internet

It isn't even close to 1% of the internet, much less virtually the entire internet. According to the latest dump, Common Crawl has 4.3B pages, but Google in 2016 estimated there are 130T pages. The difference between 130T and 4.3B is about 130T. Even if you narrow it down to Google's searchable text index it's "100's of billions of pages" and roughly 100P compared to CommonCrawl's 400T.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#45

Post by PaulHoule »

taneq wrote: post #40
PaulHoule wrote: post #36

Earlier quoted context omitted.

I think so. It’s hard for me to believe that the decision surfaces inside those models are really curved enough (like the folds of your brain) to really take advantage of FP32 numbers inside vectors: that is I just don’t believe it is x = 0 means “fly” x = 0.01 means “drive” x = 0.02 means “purple” but rather more like x 1.5 means “hot” which is one reason why quantization (often 1 bit) works. Also it is a reason why…

Are you conflating nonlinear embedding spaces with the physical curvature of the cerebellum? I don't think there's a direct mapping.

My mental picture is that violently curved decision surfaces could look like the convolutions of the brain even though they have nothing to do with how the brain actually works.

I think of how tSNE and other algorithms sometimes produce projections that sometimes look like that (maybe that’s just what you get when you have to bend something complicated to fit into a 2-d space) and frequently show cusps that to me look like a sign of trouble (took me a while in my PhD work to realize how Poincaré sections from 4 or 6 dimensions can look messed up when a part of the energy surface tilts perpendicularly to the projection surface.)

I still find it hard to believe that dense vectors are the right way to deal with text despite the fact that they work so well. For images it is one thing because changing one pixel a little doesn’t change the meaning of an image, but changing a single character of a text can completely change the meaning of the text. Also there’s the reality that if you randomly stick together tokens you get something meaningless, so it seems almost all of the representation space covers ill formed texts and only a low dimensional manifold holds the well formed texts. Now the decision surfaces really have to be nonlinear and crumpled over all but I think there’s a definitely a limit on how crumpled those surfaces can be.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#46

Post by mjburgess »

numeri wrote: post #27
mjburgess wrote: post #21

Earlier quoted context omitted.

LLMs cannot determine the physical location of any atoms. they cannot plan movement, and so on. LLMs are just completing patterns of text that have been given before, 'everthing ever written' is both a lot for any individual person to read; but also, almost nothing, in that to propertly describe a table requires more information text is itself an extremely compressed medium which lacks almost any information about th…

I didn't imply that they know anything about where atoms are, I was just pointing out the sheer absurdity of that volume of data. I should make it clear that my comparison there is unfair and mostly just funny – you don't need to store every possible combination of 10 tokens, because most of them will be nonsense, so you wouldn't actually need that much storage. That being said, it's been fairly solidly proven that L…

> fairly solidly proven that LLMs aren't just lookup tables/stochastic parrots

Well i'd strongly disagree. I see no evidence of this; I'm am quite well acquainted with the literature.

All empirical statistical AI is just a means of approximating an empirical distribution. The problem with NLP is that there is no empirical function from text tokens to meanings; just as there is no function from sets of 2D images to a 3D structure.

We know before we start that the distributions of text tokens are only coincidentally related to the distributions of meanings. The question is just how much value that coincidence has in any given task.

(Consider, eg., that if I ask, "do you like what i'm wearing?" there is no distribution of responses which is correct. I do not want you to say "yes" 99/100, or even 100/100 times. etc. what I want you to say is a word caused a mental state you have: that of (dis)liking what i'm wearing.

Since no statistical AI systems generate outputs based on causal features of reality, we know a priori that almost all possible questions that can be asked cannot be answered by LLMs.

They are only useful where questions have cannonical answers; and only because "cannonical" means that a text->text function is likely to be conidentally indistinguishable from a the meaning->meaning function we're interested in).

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#47

Post by Y_Y »

PaulHoule wrote: post #45
taneq wrote: post #40

Earlier quoted context omitted.

Are you conflating nonlinear embedding spaces with the physical curvature of the cerebellum? I don't think there's a direct mapping.

My mental picture is that violently curved decision surfaces could look like the convolutions of the brain even though they have nothing to do with how the brain actually works. I think of how tSNE and other algorithms sometimes produce projections that sometimes look like that (maybe that’s just what you get when you have to bend something complicated to fit into a 2-d space) and frequently show cusps that to me loo…

This is interesting. It makes me think of an "immersion"[0], as in a generalization of the concept of "embedding" in differential geometry.

I share your uneasiness about mapping words to vectors and agree that it feels as if we're shoehorning some more complex space into a computationally convenient one.

[0] https://en.wikipedia.org/wiki/Immersion_(mathematics)

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#48

Post by retrofrost »

This is amazing work, but to me it highlights some of the biggest problems in the current AI zeitgeist, we are not really trying to work on any neuron or ruleset that isnt much different from the perceptron thats just a sumnation function. Is it really that suprising that we just see this same structure repeated in the models. Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? We have all sorts of unique training methods and encoding schemes that don't ever get used because the big libraries don't support them. Until, we start seeing real varation in the fundamental rulesets of neuralnets we are always just going to be fighting against the fact these are just perceptrons with extra steps.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#49

Post by ramses0 »

bionhoward wrote: post #41
ramses0 wrote: post #22

Earlier quoted context omitted.

Is there some sort of "LLM-on-Wikipedia" competition? ie: given "just wikipedia" what's the best score people can get on however these models are evaluated. I know that all the commercial ventures have a voracious data-input set, but it seems like there's room for dictionary.llm + wikipedia.llm + linux-kernel.llm and some sort of judging / bake-off for their different performance capabilities. Or does the training tr…

Yes, that’s known as the Hutter Prize http://prize.hutter1.net/

Not exactly, because LLM's seem to be exhibiting value via "lossy knowledge response" vs. "exact reproduction measured in bytes", but close.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#50

Post by wongarsu »

Animats wrote: post #30
mike_hearn wrote: post #19

This is really cool. My mind goes immediately to what sort of functions are being used to encode programming knowledge, and if they are also simple linear functions whether the standard library or other libraries can be directly uploaded into an LLMs brain as it evolves, without needing to go through a costly training or performance-destroying fine-tune. That's still a sci-fi ability today but it seems to be getting…

That's a good point. It may be possible to directly upload predicate-type info into a LLM. This could be especially useful if you need to encode tabular data. Somewhere, someone probably read this and is thinking about how to export Excel or databases to an LLM. It's encouraging to see people looking inside the black box successfully. The other big result in this area was that paper which found a representation of a…

The opposite is also exciting: build a loss function that punishes models for storing knowledge. One of the issues of current models is that they seem to favor lookup over reasoning. If we can punish models (during training) for remembering that might cause them to become better at inference and logic instead.
Post reply on HN