Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#91

Post by _3u10 »

jedberg wrote: post #76
inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

> and biologically implausible I really like this approach. Showing that we must be doing it wrong because our brains are more efficient and we aren't doing it like our brains. Is this a common thing in ML papers or something you came up with?

Nah it’s just physics, it’s like wheels being more efficient than legs.

We know there is a more efficient solution (human brain) but we don’t know how to make it.

So it stands to reason that we can make more efficient LLMs, just like a CPU can add numbers more efficiently than humans.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#92

Post by Ericson2314 »

vessenes wrote: post #34
refulgentis wrote: post #25

Earlier quoted context omitted.

Yes. ( https://news.ycombinator.com/item?id=36425375 , believe there's been more) There's a quite intense backlog of new stuff that hasn't made it to prod. (I would have told you in 2023 that we would have ex. switched to Mamba-like architectures in at least one leading model) Broadly, it's probably unhelpful that: - absolutely no one wants the PR of releasing a model that isn't competitive with the latest peers - ab…

For sure read Stephenson’s essay on path dependence; it lays out a lot of these economic and social dynamics. TLDR - we will need a major improvement to see something novel pick up steam most likely.

Yeah everyone spending way to much money in things we barely understand is a recipe for insane path dependence.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#93

Post by artificialprint »

janalsncm wrote: post #67
thewarrior wrote: post #42

Earlier quoted context omitted.

Any chance that “reasoning” can fix this

It kind of depends. You can broadly call any kind of search “reasoning”. But search requires 1) enumerating your possible options and 2) assigning some value to those options. Real world problem solving makes both of those extremely difficult. Unlike in chess, there’s a functionally infinite number of actions you can take in real life. So just argmax over possible actions is going to be hard. Two, you have to have so…

Isn't something about alphago also involves "infinitely" many possible outcomes? Yet they cracked it, right?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#94

Post by leereeves »

esafak wrote: post #84
parsimo2010 wrote: post #81

Earlier quoted context omitted.

I don't think GP was implying that brains are the optimum solution. I think you can interpret GP's comments like this- if our brains are more efficient than LLMs, then clearly LLMs aren't optimally efficient. We have at least one data point showing that better efficiency is possible, even if we don't know what the optimal approach is.

I agree. Spiking neural networks are usually mentioned in this context, but there is no hardware ecosystem behind them that can compete with Nvidia and CUDA.

Investments in AI are now counting by billions of dollars. Would that be enough to create an initial ecosystem for a new architecture?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#95

Post by eximius »

I believe that so long as weights are fixed at inference time, we'll be at a dead end.

Will Titans be sufficiently "neuroplastic" to escape that? Maybe, I'm not sure.

Ultimately, I think an architecture around "looping" where the model outputs are both some form of "self update" and "optional actionality" such that interacting with the model is more "sampling from a thought space" will be required.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#96

Post by simne »

I'm not deep researcher, more like amateur, but could explain some things.

Most problem with current approach, to grow abilities, need to add more neurons, but this is not just energy consuming, but also knowledge consuming, mean, at GPT-4 level all text sources of humanity already exhausted and model become essentially overfitted. So looks like multi-modal models appear not because so good, but because they could learn on additional sources (audio/video).

I seen few approaches to overcome problem of overfitting, but as I understand not exist universal solution.

For example, tried approach to create from current texts some synthetic training data, but this idea is limited by definition.

So, current LLMs appear to hit dead end, and researchers now trying to find exit from this dead end. I believe, nearest years somebody will invent some universal solution (probably, complex of approaches) or suggest another architecture, and progress of AI will continue.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#97

Post by ortsa »

refulgentis wrote: post #25
ortsa wrote: post #20

Earlier quoted context omitted.

Has anybody ever messed with adding a "backspace" token?

Yes. ( https://news.ycombinator.com/item?id=36425375 , believe there's been more) There's a quite intense backlog of new stuff that hasn't made it to prod. (I would have told you in 2023 that we would have ex. switched to Mamba-like architectures in at least one leading model) Broadly, it's probably unhelpful that: - absolutely no one wants the PR of releasing a model that isn't competitive with the latest peers - ab…

Oh yeah, that's exactly what I was thinking of! Seems like it would be very useful for expert models with domains with more definite "edges" (if I'm understanding it right)

As for the fragmentation of progress, I guess that's just par the course for any tech with a such a heavy private/open source split. It would take a huge amount of work to trawl through this constant stream of 'breakthroughs' and put them all together.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#98

Post by Horffupolde »

__rito__ wrote: post #14

Sligtly related: Energy Based Models (EBMs) are better in theory and yet too resource intensive. I tried to sell using EBMs to my org, but the price for even a small use case was prohibitive. I learned it from: https://youtube.com/playlist?list=PLLHTzKZzVU9eaEyErdV26ikyo... Yann LeCun, and Michael Bronstein and his colleagues have some similarities in trying to properly Sciencify Deep Learning. Yann LeCun's approach,…

WTF. The cardinality of words is 100,000.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#99

Post by giantg2 »

I feel like some hallucinations aren't bad. Isn't that basically what a new idea is - a hallucination of what could be? The ability to come up with new things, even if they're sometimes wrong, can be useful and happen all the time with humans.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#100

Post by hn_user82179 »

giantg2 wrote: post #99

I feel like some hallucinations aren't bad. Isn't that basically what a new idea is - a hallucination of what could be? The ability to come up with new things, even if they're sometimes wrong, can be useful and happen all the time with humans.

That’s a really interesting thought. I think the key part (as a consumer of AI tools) would be identifying the things that are guesses vs deductions vs complete accurate based on the training data. I would happily look up or think about the output parts that are possibly hallucinated myself but we don’t currently get that kind of feedback. Whereas a human could list things out that they know, and then highlight the things they making educated guesses about, which makes it easier to build upon.
Post reply on HN