Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any insider takes on Yann LeCun's push against current architectures?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 343 posts

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#81

Post by parsimo2010 »

esafak wrote: post #78
jedberg wrote: post #76

Earlier quoted context omitted.

> and biologically implausible I really like this approach. Showing that we must be doing it wrong because our brains are more efficient and we aren't doing it like our brains. Is this a common thing in ML papers or something you came up with?

Evolution does not need to converge on the optimum solution. Have you heard of https://en.wikipedia.org/wiki/Bio-inspired_computing ?

I don't think GP was implying that brains are the optimum solution. I think you can interpret GP's comments like this- if our brains are more efficient than LLMs, then clearly LLMs aren't optimally efficient. We have at least one data point showing that better efficiency is possible, even if we don't know what the optimal approach is.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#82

Post by vessenes »

inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

I’m looking forward to it! Inefficiency (if we mean energy efficiency) conceptually doesn’t bother me very much in that feels like Silicon design has a long way to go still, but I like the idea of looking at biology for both ideas and guidance.

Inefficiency in data input is also an interesting concept. It seems to me humans get more data in than even modern frontier models; if you use the gigabit/s estimates for sensory input. Care to elaborate on your thoughts?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#83

Post by vessenes »

bobosha wrote: post #71

I argue that JEPA and its Energy-Based Model (EBM) framework fail to capture the deeply intertwined nature of learning and prediction in the human brain—the “yin and yang” of intelligence. Contemporary machine learning approaches remain heavily reliant on resource-intensive, front-loaded training phases. I advocate for a paradigm shift toward seamlessly integrating training and prediction, aligning with the principle…

Thank you. So, quick q - it would make sense to me that JEPA is an outcome of the YLC work; would you say that’s the case?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#84

Post by esafak »

parsimo2010 wrote: post #81
esafak wrote: post #78

Earlier quoted context omitted.

Evolution does not need to converge on the optimum solution. Have you heard of https://en.wikipedia.org/wiki/Bio-inspired_computing ?

I don't think GP was implying that brains are the optimum solution. I think you can interpret GP's comments like this- if our brains are more efficient than LLMs, then clearly LLMs aren't optimally efficient. We have at least one data point showing that better efficiency is possible, even if we don't know what the optimal approach is.

I agree. Spiking neural networks are usually mentioned in this context, but there is no hardware ecosystem behind them that can compete with Nvidia and CUDA.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#85

Post by esafak »

bobosha wrote: post #71

I argue that JEPA and its Energy-Based Model (EBM) framework fail to capture the deeply intertwined nature of learning and prediction in the human brain—the “yin and yang” of intelligence. Contemporary machine learning approaches remain heavily reliant on resource-intensive, front-loaded training phases. I advocate for a paradigm shift toward seamlessly integrating training and prediction, aligning with the principle…

So you believe humans spend more energy on prediction, relative to computers? Isn't that because personal computers are not powerful enough to train big models, and most people have no desire to? It is more economically efficient to socialize the cost of training, as is done. Are you thinking of a distributed training, where we split the work and cost? That could happen when robots become more widespread.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#86

Post by esafak »

TrainedMonkey wrote: post #5

This is a somewhat nihilistic take with an optimistic ending. I believe humans will never fix hallucinations. Amount of totally or partially untrue statements people make is significant. Especially in tech, it's rare for people to admit that they do not know something. And yet, despite all of that the progress keeps marching forward and maybe even accelerating.

I think most humans are perfectly capable of admitting to themselves when they do not know something. Computers ought to do better.

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#87

Post by vessenes »

bobosha wrote: post #71

I argue that JEPA and its Energy-Based Model (EBM) framework fail to capture the deeply intertwined nature of learning and prediction in the human brain—the “yin and yang” of intelligence. Contemporary machine learning approaches remain heavily reliant on resource-intensive, front-loaded training phases. I advocate for a paradigm shift toward seamlessly integrating training and prediction, aligning with the principle…

Update: Interesting paper, thanks. Comment on selection for Hydra — you mention v1 uses an arithmetic mean across timescales for prediction. Taking this analogy of the longer windows encapsulating different timescales, I’d propose it would be interesting to train a layer to predict weighting of the timescale predictions. Essentially — is this a moment where I need to focus on what just happened, or is this a moment in which my long range predictions are more important?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#88

Post by vessenes »

jurschreuder wrote: post #46

This concept comes from Hopfield networks. If two nodes are on, but the connection between them is negative, this causes energy to be higher. If one of those nodes switches off, energy is reduced. With two nodes this is trivial. With 10 nodes it's more difficult to solve, and with billions of nodes it is impossible to "solve". All you can do then is try to get the energy as low as possible. This way also neural netwo…

So, what’s modeled as a “node” in an EBM, and what’s modeled as a connection? Are they vectors in a tensor, (well I suppose almost certainly that’s a yes). Do they run side by side a model that’s being trained? Is the node connectivity architecture fixed or learned?

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#89

Post by snats »

Not an insider but imo the work on diffusion language models like LLaDA is really exciting. It's pretty obvious that LLMs are good but they are pretty slow. And in a world where people want agents you want a lot of the time something that might not be that smart but is capable of going really fast + searches fast. You only need to solve search in a specific domain for most agents. You don't need to solve the entire knowledge of human history in a single set of weights

Re: Ask HN: Any insider takes on Yann LeCun's push against current architectures?

#90

Post by snowwrestler »

inimino wrote: post #75

I have a paper coming up that I modestly hope will clarify some of this. The short answer should be that it's obvious LLM training and inference are both ridiculously inefficient and biologically implausible, and therefore there has to be some big optimization wins still on the table.

I think the hard question is whether those wins can be realized with less effort than what we’re already doing, though.

What I mean is this: A brain today is obviously far more efficient at intelligence than our current approaches to AI. But a brain is a highly specialized chemical computer that evolved over hundreds of millions of years. That leaves a lot of room for inefficient and implausible strategies to play out! As long as wins are preserved, efficiency can improve this way anyway.

So the question is really, can we short cut that somehow?

It does seem like doing so would require a different approach. But so far all our other approaches to creating intelligence have been beaten by the big simple inefficient one. So it’s hard to see a path from here that doesn’t go that route.

Post reply on HN