Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#171

Post by wegfawefgawefg »

smelendez wrote: post #169
pfisch wrote: post #167

Earlier quoted context omitted.

When you talk to ~3 year old children they hallucinate quite a lot. Really almost nonstop when you ask them about almost anything. I'm not convinced that what LLM's are doing is that far off the beaten path from our own cognition.

That’s interesting. Lots of modern kids probably get exposed to way more fiction than fact thanks to TV. I was an only child and watched a lot of cartoons and bad sitcoms as a kid, and I remember for a while my conversational style was way too full of puns, one-liners, and deliberately naive statements made for laughs.

i wish more people were still like that

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#172

Post by refulgentis »

nextts wrote: post #170
pfisch wrote: post #167

Earlier quoted context omitted.

When you talk to ~3 year old children they hallucinate quite a lot. Really almost nonstop when you ask them about almost anything. I'm not convinced that what LLM's are doing is that far off the beaten path from our own cognition.

Interesting but a bit non-sequitur. Humans learn and get things wrong. A formative mind is a seperate subject. But a 3 year old is vastly intelligent vs an LLM. Comparing the sounds from a 3 year old and the binary tokens from an LLM is simply indulging the illusion. I am also not convinced that magicians saw people in half, and thise people survive, defying medical and physical science.

I'm not sure I buy that, I didnt find the counter argument persuasive, but this comment basically took you from thoughtful to smug — unfairly so, ironically, because I've been so bored by not understanding Yann's "average housecat is smarter than an LLM"

Speaking of which...I'm glad you're here ,because I have an interlocutor I can be honest with while getting at the root question of the Ask HN.

What in the world does it mean that a 3 year old is smarter than an LLM?

I don't understand the thing about sounds vs. binary either. Like, both go completely over my head.

The only thing I can think of it's some implied intelligence scoring index where "writing a resume" and "writing creative fiction" and "writing code" are in the same bucket thats limited to 10 points. Then there's anther 10 point bucket for "can vocalize", that an LLM is going to get 0 on.*

If that's the case, it comes across as intentionally obtuse, in that there's an implied prior about how intelligence is scored and it's a somewhat unique interpretation that seems more motivated by the question than reflective of reality — i.e. assume a blind mute human who types out answers out that match our LLMs. Would we say that person is not as intelligent as a 3 year old?

* well, it shouldn't, but for now let's bypass that quagmire

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#174

Post by nsonha »

unsupp0rted wrote: post #69
tyronehed wrote: post #26

Any transformer based LLM will never achieve AGI because it's only trying to pick the next word. You need a larger amount of planning to achieve AGI. Also, the characteristics of LLMs do not resemble any existing intelligence that we know of. Does a baby require 2 years of statistical analysis to become useful? No. Transformer architectures are parlor tricks. They are glorified Google but they're not doing anything o…

> Does a baby require 2 years of statistical analysis to become useful? Well yes, actually.

of the entire human race's knowledge, and it's like from written history, not 2 years ago.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#175

Post by fovc »

I wonder if the error propagation problem could be solved with a “branching” generator? Basically at every token you fork off N new streams, with some tree pruning policy to avoid exponential blowup. With a bit of bookkeeping you could make an attention mask to support the parallel streams in the same context sharing prefixes. Perhaps that would allow more of an e2e error minimization than the greedy generation algorithm in use today?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#176

Post by the13 »

Matthyze wrote: post #11
jawiggins wrote: post #4

I'm not an ML researcher, but I do work in the field. My mental model of AI advancements is that of a step function with s-curves in each step [1]. Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. Examples of steps…

> Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. That seems to be how science works as a whole. Long periods of little progress between productive paradigm shifts.

[dead]

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#178

Post by ahazred8ta »

Matthyze wrote: post #11
jawiggins wrote: post #4

I'm not an ML researcher, but I do work in the field. My mental model of AI advancements is that of a step function with s-curves in each step [1]. Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. Examples of steps…

> Each time there is an algorithmic advancement, people quickly rush to apply it to both existing and new problems, demonstrating quick advancements. Then we tend to hit a kind of plateau for a number of years until the next algorithmic solution is found. That seems to be how science works as a whole. Long periods of little progress between productive paradigm shifts.

It's been described as fumbling around in a dark room until you find the light switch. At which point you can see the doorway leading to the next dark room.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#179

Post by nonagono »

Many of his arguments make “logical” sense, but one way to evaluate them is: would they have applied equally well 5 years ago? and would that have predicted LLMs will never write (average) poetry, or solve math, or answer common-sense questions about the physical world reasonably well? Probably. But turns out scale is all we needed. So yeah, maybe this is the exact point where scale stops working and we need to drastically change architectures. But maybe we just need to keep scaling.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#180

Post by nextts »

refulgentis wrote: post #172
nextts wrote: post #170

Earlier quoted context omitted.

Interesting but a bit non-sequitur. Humans learn and get things wrong. A formative mind is a seperate subject. But a 3 year old is vastly intelligent vs an LLM. Comparing the sounds from a 3 year old and the binary tokens from an LLM is simply indulging the illusion. I am also not convinced that magicians saw people in half, and thise people survive, defying medical and physical science.

I'm not sure I buy that, I didnt find the counter argument persuasive, but this comment basically took you from thoughtful to smug — unfairly so, ironically, because I've been so bored by not understanding Yann's "average housecat is smarter than an LLM" Speaking of which...I'm glad you're here ,because I have an interlocutor I can be honest with while getting at the root question of the Ask HN. What in the world doe…

It is easy to cross wires in a HN thread.

I think what makes this discussion hard (hell it would be a hard PhD topic!) is:

What do we mean by smart? Intelligent? Etc.

What is my agenda and what is yours? What are we really asking?

I won't make any more arguments but pose these questions. Not for you to answer but everyone to think about:

Given (assuming) mammals including us have evolved and developed thought and language as a survival advantage, and LLMs use language because they have been trained on text produced by humans (as well as RLHF) - how do we tell on the scale of "Search engine for human output" to "Conscious Intelligent Thinking Being" where the LLM fits?

When a human says I love you, do they mean it, or is it merely 3 tokens? If an LLM says it, does it mean it?

I think the 3yr old thing is a red herring because adult intelligence VS AI is hard enough to compare (and we are the adults!) let alone bring children brain development into it. LLMs do not self organise their hardware. I'd say forget about 3 year olds for now. Talk about adults brainfarts instead. They happen!

Post reply on HN