Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

news.mit.edu

Post reply on HN
Print view View original

101–110 of 156 posts

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#101

Post by Tijdreiziger »

gyrovagueGeist wrote: post #98
erisinger wrote: post #76

Earlier quoted context omitted.

Not a personal attack. These posters are smarter than I am, just ribbing them about misusing the terminology. "Maxima" is plural, "maximum" is singular. So you would say "a local maximum," or "several local maxima." Not "a local maxima" or, the one that really got me, "getting trapped in local maxima's." As for the rest of it, carry on. Good discussion.

While "local maximas" is wrong, I think "a local maxima" is a valid way to say "a member of the set of local maxima" regardless of the number of elements in the set. It could even be a singleton.

You can't have one maxima in the same way you can't have one pencils. That's just how English works.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#102

Post by MichaelZuo »

HarHarVeryFunny wrote: post #55
MuffinFlavored wrote: post #10

Earlier quoted context omitted.

say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.) the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" Is that a good summary? > The model uses its lear…

> the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" > Is that a good summary? No - there's a lot more going on. It's not just mapping input patterns to output patterns. A good starting point to understand it are linguist's sentence-structure trees (and these were the inspiration for the "transformer"…

Thanks for the explanation.

Since unicode has well over 64000 symbols, does that imply models, trained on a large corpus, must necessarily have at least 64000 ‘branches’ at the bottom layer?

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#103

Post by refulgentis »

foobiekr wrote: post #78
visarga wrote: post #51

Earlier quoted context omitted.

> Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.

"won" They barely work for a lot of cases (i.e., anything where accuracy matters, despite the bubble's wishful thinking). It's likely that something will sunset them in the next few years.

It seems cloyingly performative grumpy old man once you're at "it barely works and it's a bubble and blah blah" in response to a discussion about their comparative advantage (yeah, they won, and absolutely convincingly so)

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#104

Post by refulgentis »

dartos wrote: post #52
visarga wrote: post #51

Earlier quoted context omitted.

> Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.

I mean RWKV seems promising and isn’t a transformer model. Transformers have first mover advantage. They were the first models that scaled to large parameter counts. That doesn’t mean they’re the best or that they’ve won, just that they were the first to get big (literally and metaphorically)

It doesn't seem promising, a one man band has been doing a quixotic quest based on intuition and it's gotten ~nowhere, and it's not for lack of interest in alternatives. There's never been a better time to have a different approach - is your metric "times I've seen it on HN with a convincing argument for it being promising?" -- I'm not embarrassed to admit that is/was mine, but alternatively, you're aware of recent breakthroughs I haven't seen.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#105

Post by pixl97 »

Tijdreiziger wrote: post #101
gyrovagueGeist wrote: post #98

Earlier quoted context omitted.

While "local maximas" is wrong, I think "a local maxima" is a valid way to say "a member of the set of local maxima" regardless of the number of elements in the set. It could even be a singleton.

You can't have one maxima in the same way you can't have one pencils. That's just how English works.

You can't have one local maxima, it would be the global maxima. So by saying local maxima you're assuming the local is just a piece of a larger whole, even if that global state is otherwise undefined.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#106

Post by 6gvONxR4sf7o »

xanderlewis wrote: post #89
ldjkfkdsjnv wrote: post #66

Earlier quoted context omitted.

Cannot understand people claiming we are in a local maxima, when we literally had an ai scientific breakthrough only in the last two years.

Which breakthrough in the last two years are you referring to?

If you had to reduce it to one thing, it's probably that language models are capable few shot and zero shot learners. In other words, training a model to simply predict the next word on naturally occurring text, you end up with an tool you can use for generic tasks, roughly speaking.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#107

Post by Retric »

victorbjorklund wrote: post #87
foobiekr wrote: post #78

Earlier quoted context omitted.

"won" They barely work for a lot of cases (i.e., anything where accuracy matters, despite the bubble's wishful thinking). It's likely that something will sunset them in the next few years.

That is how evolution works. Something wins until something else comes along and win. And so on forever.

Evolution generally favors multiple winners in different roles over a single dominate strategy.

People tend to favor single winners.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#108

Post by samus »

haltIncomplete wrote: post #73
gwervc wrote: post #60

Earlier quoted context omitted.

"We" are not forbidding you to open a computer, start experimenting and publishing some new method. If you're so convinced that "we" are stuck in a local maxima, you can do some of the work you are advocating instead of asking other to do it for you.

All we’re doing is engineering new data compression and retrieval techniques: https://arxiv.org/abs/2309.10668 Are we sure there’s anything “net new” to find within the same old x86 machines, within the same old axiomatic systems of the past? Math is a few operations applied to carving up stuff and we believe we can do that infinitely in theory. So “all math that abides our axiomatic underpinnings” is valid regardles…

If you abstract far enough then yes, everything what we are doing is somehow akin to what we have done before. But that then also applies to what Einstein has done.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#109

Post by samus »

retrofrost wrote: post #57
visarga wrote: post #51

Earlier quoted context omitted.

> Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.

Just because papers are getting published doesn't mean its actually gaining any traction. I mean we have known that time series of signals recieves plays a huge role in how bio neurons functionally operate and yet we have nearly no examples of spiking networks being pushed beyond basic academic exploration. We have known glial cells play a critical role in biological neural and yet you can probably count the number o…

Who said that we peaked with transformers? I sure hope we did not. The current focus on them is just institutional inertia. Worst case another AI winter comes, at the end of which a newer, more promising technology would manage to attract funding anew.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#110

Post by samus »

derefr wrote: post #9

Help me understand: when they say that the facts are stored as a linear function… are they saying that the LLM has a sort of N-dimensional “fact space” encoded into the model in some manner, where facts are embedded into the space as (points / hyperspheres / Voronoi manifolds / etc); and where recalling a fact is — at least in an abstract sense — the NN computing / remembering a key to use, and then doing a key-value…

The memory palace is a hack that works because in an evolutionary sense our brain's purpose is to help us navigate our world and be effective in it. To do that, it has to be really good at remembering locations, to plot paths through and between them, and to translate that into speech or motion.
Post reply on HN