Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#101

Post by janalsncm »

artificialprint wrote: post #93
janalsncm wrote: post #67

Earlier quoted context omitted.

It kind of depends. You can broadly call any kind of search “reasoning”. But search requires 1) enumerating your possible options and 2) assigning some value to those options. Real world problem solving makes both of those extremely difficult. Unlike in chess, there’s a functionally infinite number of actions you can take in real life. So just argmax over possible actions is going to be hard. Two, you have to have so…

Isn't something about alphago also involves "infinitely" many possible outcomes? Yet they cracked it, right?

Go is played on a 19x19 board. At the beginning of the game the first player has 361 possible moves. The second player then has 360 possible moves. There is always a finite and relatively “small” number of options.

I think you are thinking of the fact that it had to be approached in a different way than Minimax in chess because a brute force decision tree grows way too fast to perform well. So they had to learn models for actions and values.

In any case, Go is a perfect information game, which as I mentioned before, is not the same as problems in the real world.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#103

Post by mft_ »

eximius wrote: post #95

I believe that so long as weights are fixed at inference time, we'll be at a dead end. Will Titans be sufficiently "neuroplastic" to escape that? Maybe, I'm not sure. Ultimately, I think an architecture around "looping" where the model outputs are both some form of "self update" and "optional actionality" such that interacting with the model is more "sampling from a thought space" will be required.

Very much this. I’ve been wondering why I’ve not seen it much discussed.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#104

Post by sockaddr »

snowwrestler wrote: post #90
inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

I think the hard question is whether those wins can be realized with less effort than what we’re already doing, though. What I mean is this: A brain today is obviously far more efficient at intelligence than our current approaches to AI. But a brain is a highly specialized chemical computer that evolved over hundreds of millions of years. That leaves a lot of room for inefficient and implausible strategies to play ou…

Also, a brain evolved to be a stable compute platform in body that finds itself in many different temperature and energy regimes. And the brain can withstand and recover from some pretty severe damage. So I'd suspect an intelligence that is designed to run in a tighter temp/power envelope with no need for recovery or redundancy could be significantly more efficient than our brain.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#105

Post by jiggawatts »

My observation from the outside watching this all unfold is that not enough effort seems to be going into the training schedule.

I say schedule because the “static data once through” is the root of the problem in my mind is one of the root problems.

Think about what happens when you read something like a book. You’re not “just” reading it, you’re also comparing it to other books, other books by the same author, while critically considering the book recommendations made by your friend. Any events in the book get compared to your life experience, etc…

LLM training does none of this! It’s a once-through text prediction training regime.

What this means in practice is that an LLM can’t write a review of a book unless it has read many reviews already. They have, of course, but the problem doesn’t go away. Ask an AI to critique book reviews and it’ll run out of steam because it hasn’t seen many of those. Critiques of critiques is where they start falling flat on their face.

This kind of meta-knowledge is precisely what experts accumulate.

As a programmer I don’t just regurgitate code I’ve seen before with slight variations — instead I know that mainstream criticisms of micro services misses their key benefit of extreme team scalability!

This is the crux of it: when humans read their training material they are generating an “n+1” level in their mind that they also learn. The current AI training setup trains the AI only the “n”th level.

This can be solved by running the training in a loop for several iterations after base training. The challenge of course is to develop a meaningful loss function.

IMHO the “thinking” model training is a step in the right direction but nowhere near enough to produce AGI all by itself.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#106

Post by [deleted] »

snowwrestler wrote: post #90
inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

I think the hard question is whether those wins can be realized with less effort than what we’re already doing, though. What I mean is this: A brain today is obviously far more efficient at intelligence than our current approaches to AI. But a brain is a highly specialized chemical computer that evolved over hundreds of millions of years. That leaves a lot of room for inefficient and implausible strategies to play ou…

[deleted]

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#107

Post by fallingknife »

sockaddr wrote: post #104
snowwrestler wrote: post #90

Earlier quoted context omitted.

I think the hard question is whether those wins can be realized with less effort than what we’re already doing, though. What I mean is this: A brain today is obviously far more efficient at intelligence than our current approaches to AI. But a brain is a highly specialized chemical computer that evolved over hundreds of millions of years. That leaves a lot of room for inefficient and implausible strategies to play ou…

Also, a brain evolved to be a stable compute platform in body that finds itself in many different temperature and energy regimes. And the brain can withstand and recover from some pretty severe damage. So I'd suspect an intelligence that is designed to run in a tighter temp/power envelope with no need for recovery or redundancy could be significantly more efficient than our brain.

The brain only operates in a very narrow temperature range too. 5 degrees C in either direction from 37 and you're in deep trouble.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#108

Post by esafak »

leereeves wrote: post #94
esafak wrote: post #84

Earlier quoted context omitted.

I agree. Spiking neural networks are usually mentioned in this context, but there is no hardware ecosystem behind them that can compete with Nvidia and CUDA.

Investments in AI are now counting by billions of dollars. Would that be enough to create an initial ecosystem for a new architecture?

Nvidia has a big lead, and hardware is capital intensive. I guess an alternative would make sense in the battery-powered regime, like robotics, where Nvidia's power hungry machines are at a disadvantage. This is how ARM took on Intel.
Post reply on HN