Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

news.mit.edu

Post reply on HN
Print view View original

71–80 of 156 posts

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#71

Post by wanderingbort »

nicklecompte wrote: post #68
visarga wrote: post #51

Earlier quoted context omitted.

> Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.

His point is that "evolution by selection" also includes that transformers are easy to implement with modern linear algebra libraries and cheap to scale on current silicon, both of which are engineering details with no direct relationship to their innate efficacy at learning (though indirectly it means you scale up the training data for more inefficient learning).

I think it is correct to include practical implementation costs in the selection.

Theoretical efficacy doesn’t guarantee real world efficacy.

I accept that this is self reinforcing but I favor real gains today over potentially larger gains in a potentially achievable future.

I also think we are learning practical lessons on the periphery of any application of AI that will apply if a mold-breaking solution becomes compelling.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#72

Post by vlovich123 »

erisinger wrote: post #69
Kerb_ wrote: post #62

Earlier quoted context omitted.

You can think chemotherapy is a local maxima for cancer treatment and hope medical research seeks out other options without having the resources to do it yourself. Not all of us have access to the tools and resources to start experimenting as casually as we wish we could.

Not a single one of you bigbrains used the word "maxima" correctly and it's driving me crazy.

As I understand it a local maxima means you’re at a local peak but there may be higher maximums elsewhere. As I read it, transformers are a local maximum in the sense of outperforming all other ML techniques as the AI technique that gets the closest to human intelligence.

Can you help my little brain understand the problem by elaborating?

Also you may want to chill with the personal attacks.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#73

Post by haltIncomplete »

gwervc wrote: post #60
retrofrost wrote: post #57

Earlier quoted context omitted.

Just because papers are getting published doesn't mean its actually gaining any traction. I mean we have known that time series of signals recieves plays a huge role in how bio neurons functionally operate and yet we have nearly no examples of spiking networks being pushed beyond basic academic exploration. We have known glial cells play a critical role in biological neural and yet you can probably count the number o…

"We" are not forbidding you to open a computer, start experimenting and publishing some new method. If you're so convinced that "we" are stuck in a local maxima, you can do some of the work you are advocating instead of asking other to do it for you.

All we’re doing is engineering new data compression and retrieval techniques: https://arxiv.org/abs/2309.10668

Are we sure there’s anything “net new” to find within the same old x86 machines, within the same old axiomatic systems of the past?

Math is a few operations applied to carving up stuff and we believe we can do that infinitely in theory. So “all math that abides our axiomatic underpinnings” is valid regardless if we “prove it” or not.

Physical space we can exist in, a middle ground of reality we evolved just so to exist in, seems to be finite; I can’t just up and move to Titan or Mars. So our computers are coupled to the same constraints of observation and understanding as us.

What about daily life will be upended reconfirming decades old experiment? How is this not living in sunk cost fallacy?

When all you have is a hammer…

I’m reminded of Einstein’s quote about insanity.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#74

Post by dartos »

tkellogg wrote: post #54
dartos wrote: post #52

Earlier quoted context omitted.

I mean RWKV seems promising and isn’t a transformer model. Transformers have first mover advantage. They were the first models that scaled to large parameter counts. That doesn’t mean they’re the best or that they’ve won, just that they were the first to get big (literally and metaphorically)

Yeah, I'd argue that transformers created such capital saturation that there's a ton of opportunity for alternative approaches to emerge.

Speak of the devil. Jamba just hit the front page.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#75

Post by leoc »

retrofrost wrote: post #57
visarga wrote: post #51

Earlier quoted context omitted.

> Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.

Just because papers are getting published doesn't mean its actually gaining any traction. I mean we have known that time series of signals recieves plays a huge role in how bio neurons functionally operate and yet we have nearly no examples of spiking networks being pushed beyond basic academic exploration. We have known glial cells play a critical role in biological neural and yet you can probably count the number o…

(The singulars are ‘maximum’ and ‘minimum’, ‘maxima’ and ‘minima’ are the plurals.)

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#76

Post by erisinger »

vlovich123 wrote: post #72
erisinger wrote: post #69

Earlier quoted context omitted.

Not a single one of you bigbrains used the word "maxima" correctly and it's driving me crazy.

As I understand it a local maxima means you’re at a local peak but there may be higher maximums elsewhere. As I read it, transformers are a local maximum in the sense of outperforming all other ML techniques as the AI technique that gets the closest to human intelligence. Can you help my little brain understand the problem by elaborating? Also you may want to chill with the personal attacks.

Not a personal attack. These posters are smarter than I am, just ribbing them about misusing the terminology.

"Maxima" is plural, "maximum" is singular. So you would say "a local maximum," or "several local maxima." Not "a local maxima" or, the one that really got me, "getting trapped in local maxima's."

As for the rest of it, carry on. Good discussion.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#77

Post by [deleted] »

erisinger wrote: post #69
Kerb_ wrote: post #62

Earlier quoted context omitted.

You can think chemotherapy is a local maxima for cancer treatment and hope medical research seeks out other options without having the resources to do it yourself. Not all of us have access to the tools and resources to start experimenting as casually as we wish we could.

Not a single one of you bigbrains used the word "maxima" correctly and it's driving me crazy.

[deleted]

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#78

Post by foobiekr »

visarga wrote: post #51
retrofrost wrote: post #48

This is amazing work, but to me it highlights some of the biggest problems in the current AI zeitgeist, we are not really trying to work on any neuron or ruleset that isnt much different from the perceptron thats just a sumnation function. Is it really that suprising that we just see this same structure repeated in the models. Just because feedforward topologies with single neuron steps are the easiest to train and r…

> Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.

"won"

They barely work for a lot of cases (i.e., anything where accuracy matters, despite the bubble's wishful thinking). It's likely that something will sunset them in the next few years.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#79

Post by ikkiew »

retrofrost wrote: post #48

This is amazing work, but to me it highlights some of the biggest problems in the current AI zeitgeist, we are not really trying to work on any neuron or ruleset that isnt much different from the perceptron thats just a sumnation function. Is it really that suprising that we just see this same structure repeated in the models. Just because feedforward topologies with single neuron steps are the easiest to train and r…

> the perceptron thats just a sumnation[sic] function

What would you suggest?

My understanding of part of the whole NP-Complete thing is that any algorithm in the complexity class can be reduced to, among other things, a 'summation function'.

Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

#80

Post by leobg »

numeri wrote: post #11
leobg wrote: post #2

> In one experiment, they started with the prompt “Bill Bradley was a” and used the decoding functions for “plays sports” and “attended university” to see if the model knows that Sen. Bradley was a basketball player who attended Princeton. Why not just change the prompt? Name, University attended, Sport played Bill Bradley,

This is research, trying to understand the fundamentals of how these models work. They weren't actually trying to find out where Bill Bradley went to university.

Of course. But weren’t they trying to find out whether or not that fact was represented in the model’s parameters?
Post reply on HN