Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPU-rich labs have won: What's left for the rest of us is distillation

inference.net

Post reply on HN
Print view View original

11–20 of 53 posts

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#11

Post by lazide »

michelpp wrote: post #7
ilaksh wrote: post #4

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot o…

Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind…

Memristors have been tried for literally decades.

If the posters other guesses pay out the same rate, this will likely play out never.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#12

Post by hnuser123456 »

ilaksh wrote: post #4

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot o…

>memory-centric compute

This already exists: https://www.cerebras.ai/chip

They claim 44 GB of SRAM at 21 PB/s.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#13

Post by marcosdumay »

ilaksh wrote: post #4

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot o…

Memristors in particular just won't happen.

But memory-centric compute didn't happen because of Moore's law. (SNNs have the problem that we don't actually know how to use them.) Now that it's gone, it may have a chance, but it still takes a large amount of money thrown into the idea and the people with money are so risk-adverse that they create entire new risks for themselves.

Forward neural networks were very lucky that there existed a mainstream use for the kind of hardware it needed.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#14

Post by YetAnotherNick »

Deepseek main run costed $6M. qwen3-30b-a3b probably would cost few $100Ks, which is ranked 13th.

GPU cost of the final model training isn't the biggest chunk of the cost and you can probably replicate results of models like Llama 3 very cheaply. It's the cost of experiments, researchers, data collection which brings overall cost 1 or 2 order of magnitude higher.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#16

Post by ilaksh »

lazide wrote: post #11
michelpp wrote: post #7

Earlier quoted context omitted.

Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind…

Memristors have been tried for literally decades. If the posters other guesses pay out the same rate, this will likely play out never.

Other technologies tried for decades before becoming huge: Neural-network AI; Electric cars; mRNA vaccines; Solar photovoltaics; LED lighting

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#17

Post by thomassmith65 »

Perhaps one of these days a random compsci undergrad will come up a DeepSeek-calibre optimization.

Just imagine his or her 'ChatGPT with 10,000x fewer propagations' Reddit post appearing on a Monday...

...and $3 trillion of Nvidia stock going down the drain by Friday.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#18

Post by stego-tech »

BoredPositron wrote: post #9

We haven't seen a proper npu and we are in the launch of the first consumer grade unified architectures by Nvidia and AMD. The battle of homebrew AI hasn't even started yet.

Hell, we haven’t even seen actual AI yet. This is all just brute-forcing likely patterns of tokens based on a corpus of existing material, not anything brand new or particularly novel. Who would’ve guessed that giving CompSci and Mathematics researchers billions of dollars in funding and millions of GPUs in parallel without the usual constraints of government research would produce the most expensive brute-force algorithms in human history?

I still believe this is going to be an embarrassing chapter of the history of AI when we actually do create it. “Humans - with the sort of hubris only a neoliberal post-war boom period could produce - honestly thought their first serious development in computing (silicon-based mircoprocessors) would lead to Artificial General Intelligence and usher in a utopia of the masses. Instead they squandered their limited resources on a Fool’s Errand, ignoring more important crises that would have far greater impacts on their immediate prosperity in the naive belief they could create a Digital God from Silicon and Electricity alone.”

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#20

Post by therealpygon »

thomassmith65 wrote: post #17

Perhaps one of these days a random compsci undergrad will come up a DeepSeek-calibre optimization. Just imagine his or her 'ChatGPT with 10,000x fewer propagations' Reddit post appearing on a Monday... ...and $3 trillion of Nvidia stock going down the drain by Friday.

One can only hope. Maybe then they’ll sell us GPUs with 2025 quantity memory instead of 2015.
Post reply on HN