Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

311–320 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#311

Post by [deleted] »

jonplackett wrote: post #131
_3u10 wrote: post #91

Earlier quoted context omitted.

Nah it’s just physics, it’s like wheels being more efficient than legs. We know there is a more efficient solution (human brain) but we don’t know how to make it. So it stands to reason that we can make more efficient LLMs, just like a CPU can add numbers more efficiently than humans.

Wheels is an interesting analogy. Wheels are more efficient now that we have roads. But there could never have been evolutionary pressure to make them before there were roads. Wheels are also a lot easier to get to work than robotic legs and so long as there’s a road do a lot more than robotic legs.

[deleted]

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#312

Post by _3u10 »

jonplackett wrote: post #131
_3u10 wrote: post #91

Earlier quoted context omitted.

Nah it’s just physics, it’s like wheels being more efficient than legs. We know there is a more efficient solution (human brain) but we don’t know how to make it. So it stands to reason that we can make more efficient LLMs, just like a CPU can add numbers more efficiently than humans.

Wheels is an interesting analogy. Wheels are more efficient now that we have roads. But there could never have been evolutionary pressure to make them before there were roads. Wheels are also a lot easier to get to work than robotic legs and so long as there’s a road do a lot more than robotic legs.

People think the first wheel was invented for making pottery. Biological machinery for the most part has to be self-reproducing so there is a lot of limitations on design, also it has to be able to evolve, so you get inefficient solutions like the vargas nerve (i think that's its name), basically there's a really long nerve in your body that takes a route under your trachea and then back up to another part of your brain, in giraffes its something like 40 feet long to go a few inches shortest path.

Wheels other than rolling would likely never evolve naturally because there's no real incremental path from legs to wheels, where as flippers can evolve from webbed fingers incrementally getting better for moving in water.

I dunno, maybe there's an evolutionary path for wheels, but i don't think so.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#313

Post by throw310822 »

YeGoblynQueenne wrote: post #309
throw310822 wrote: post #159

Earlier quoted context omitted.

I don't get it. 1) Yes it's true, learning from text is very hard. But LLMs are multimodal now. 2) That "size of a lion" paper is from 2019, which is a geological era from now. The SOTA was GPT2 which was barely able to spit out coherent text. 3) Have you tried asking a mouse to play chess or reason its way through some physics problem or to write some code? I'm really curious in which benchmark are mice surpassing c…

Oh mice can solve a plethora of physics problems before it's time for breakfast. They have to navigate the, well, physical world, after all. I'm also really curious what benchmarks LLMs have passed that include surviving without being eaten by a cat, or a gull, or an owl, while looking for food to survive and feed one's young in an arbitrary environment chosen from urban, rural, natural etc, at random. What's ChatGPT…

> mice can solve a plethora of physics problems before it's time for breakfast

Ah really? Which ones? And nope, physical agility is not "solving a physics problem", otherwise a soccer players and figure skaters would all have PhDs, which doesn't seem to be the case.

I mean, an automated system that solves equations to keep balance is not particularly "intelligent". We usually call intelligence the ability to solve generic problems, not the ability of a very specialized system to solve the same problem again and again.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#314

Post by famouswaffles »

kryogen1c wrote: post #279
ninetyninenine wrote: post #204

Earlier quoted context omitted.

LLM is just the name. You can encode anything into the "language" including pictures video and sound.

> You can encode anything into the "language Im just a layman here, but i don't think this is true. Language is an abstraction, an interpreative mechanism of reality. A reproduction of reality, like a picture, by definition holds more information than it's abstraction does.

I think his point is that LLMs are pre-trained transformers. And pre-trained transformers are general sequence predictors. Those sequences started out as text or language only but by no means is the architecture constrained to text or language alone. You can train a transformer that embeds and predicts sound and images as well as text.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#315

Post by famouswaffles »

chongli wrote: post #296
mietek wrote: post #288

Earlier quoted context omitted.

Are LLMs having subjective experiences? Surely not. But if you claim that human subjective experiences are not the result of electrical signals in the brain, then what exactly is your position? Dualism? Personally, I think the Chinese room argument is invalid. In order for the person in the room to respond to any possible query by looking up the query in a book, the book would need to be infinite and therefore imposs…

The Chinese Room is a perfect analogy for what's going on with LLMs. The book is not infinite, it's flawed. And that's the point: we keep bumping into the rough edges of LLMs with their hallucinations and faulty reasoning because the book can never be complete. Thus we keep getting responses that make us realize the LLM is not intelligent and has no idea what it's saying. The only part where the book analogy falls do…

>The book is not infinite, it's flawed.

Oh and the human book is surely infinite and unflawed right ?

>we keep bumping into the rough edges of LLMs with their hallucinations and faulty reasoning

Both things humans also do in excess

The Chinese Room is nonsensical. Can you point to any part of your brain that understands English ? I guess you are a Chinese Room then.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#316

Post by nickpsecurity »

adamnemecek wrote: post #305
nickpsecurity wrote: post #303

Earlier quoted context omitted.

I hope you succeed. I've downloaded and at least skimmed hundreds of papers ML, many alternative architectures. A subset of them built prototypes that claimed good results on benchmarks. Of those, many didn't pass further scrutiny due to various failures. Those that did pass often failed on real-world tasks despite doing well on benchmarks. That we're so jaded by failures of published models makes us even more skepti…

How many of these were actually addressing scaling EBMs though? I'm guessing none.

Including yours. Your landing page has no architecture, model, or performance comparisons. It's non-existent. You need something more tangible for us to believe in.

Remember that scientific method requires us to reject everything by default. Only after rigorous review of a working theory or prototype do we treat it as truth. Build what you want us to believe in. Let us see it smoke the competing models of similar size in key metrics. That will do more for you than anything else.

Again, I hope you're right and I get to see energy-based models being highly competitive. I haven't.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#317

Post by YeGoblynQueenne »

throw310822 wrote: post #313
YeGoblynQueenne wrote: post #309

Earlier quoted context omitted.

Oh mice can solve a plethora of physics problems before it's time for breakfast. They have to navigate the, well, physical world, after all. I'm also really curious what benchmarks LLMs have passed that include surviving without being eaten by a cat, or a gull, or an owl, while looking for food to survive and feed one's young in an arbitrary environment chosen from urban, rural, natural etc, at random. What's ChatGPT…

> mice can solve a plethora of physics problems before it's time for breakfast Ah really? Which ones? And nope, physical agility is not "solving a physics problem", otherwise a soccer players and figure skaters would all have PhDs, which doesn't seem to be the case. I mean, an automated system that solves equations to keep balance is not particularly "intelligent". We usually call intelligence the ability to solve ge…

>> Ah really? Which ones? And nope, physical agility is not "solving a physics problem", otherwise a soccer players and figure skaters would all have PhDs, which doesn't seem to be the case.

Yes, everything that has to do with navigating physical reality, including, but not restricted to physical agility. Those are physics problems that animals, including humans, know how to solve and, very often, we have no idea how to program a computer to solve them.

And you're saying that solving physics problems means you have a PhD? So for example Archimedes did not solve any physics problems otherwise he'd have a PhD?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#318

Post by eli_gottlieb »

bravura wrote: post #132

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a br…

>(Energy minimization is a very old idea. LeCun has been on about it for a while and it's less controversial these days. Back when everyone tried to have a probabilistic interpretation of neural models, it was expensive to compute the normalization term / partition function. Energy minimization basically said: Set up a sensible loss and minimize it.)

Ehhhh, energy-based models are trained via contrastive divergence, not just minimizing a simple loss averaged over the training data.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#319

Post by bubblyworld »

chmod775 wrote: post #197
refulgentis wrote: post #172

Earlier quoted context omitted.

I'm not sure I buy that, I didnt find the counter argument persuasive, but this comment basically took you from thoughtful to smug — unfairly so, ironically, because I've been so bored by not understanding Yann's "average housecat is smarter than an LLM" Speaking of which...I'm glad you're here ,because I have an interlocutor I can be honest with while getting at the root question of the Ask HN. What in the world doe…

> What in the world does it mean that a 3 year old is smarter than an LLM? Because LLMs have terrible comprehension of the real world. Here's an example: > You: If you put a toddler next to a wig on the floor, which reaches higher? > ChatGPT: The wig would typically reach higher than the toddler, especially if the wig is a standard size or has long hair. Toddlers are generally around 2 to 3 feet tall, while wigs can…

Hah, I tried it with gpt-4o and got similarly odd results:

https://chatgpt.com/share/67d6fb93-890c-8004-909d-2bb7962c8f...

It's pretty good nonsense though. It suggests clove hitching them together, which would be a weird (and probably unsafe) thing to do even with ropes!

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#320

Post by chongli »

famouswaffles wrote: post #315
chongli wrote: post #296

Earlier quoted context omitted.

The Chinese Room is a perfect analogy for what's going on with LLMs. The book is not infinite, it's flawed. And that's the point: we keep bumping into the rough edges of LLMs with their hallucinations and faulty reasoning because the book can never be complete. Thus we keep getting responses that make us realize the LLM is not intelligent and has no idea what it's saying. The only part where the book analogy falls do…

>The book is not infinite, it's flawed. Oh and the human book is surely infinite and unflawed right ? >we keep bumping into the rough edges of LLMs with their hallucinations and faulty reasoning Both things humans also do in excess The Chinese Room is nonsensical. Can you point to any part of your brain that understands English ? I guess you are a Chinese Room then.

Humans have the ability to admit when they do not know something. We say “sorry, I don’t know, let me get back to you.” LLMs cannot do this. They either have the right answer in the book or they make up nonsense (hallucinate). And they do not even know which one they’re doing!
Post reply on HN