Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPU-rich labs have won: What's left for the rest of us is distillation

inference.net

Post reply on HN
Print view View original

21–30 of 53 posts

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#21

Post by ilaksh »

thomassmith65 wrote: post #17

Perhaps one of these days a random compsci undergrad will come up a DeepSeek-calibre optimization. Just imagine his or her 'ChatGPT with 10,000x fewer propagations' Reddit post appearing on a Monday... ...and $3 trillion of Nvidia stock going down the drain by Friday.

DeepSeek came up with several significant optimizations, not just one. And master's students do contribute to leading edge research all the time.

There have really been many significant innovations in hardware, model architecture, and software, allowing companies to keep up with soaring demand and expectations.

But that's always how it's been in high technology. You only really hear about the biggest shifts, but the optimizations are continuous.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#22

Post by ilaksh »

YetAnotherNick wrote: post #14

Deepseek main run costed $6M. qwen3-30b-a3b probably would cost few $100Ks, which is ranked 13th. GPU cost of the final model training isn't the biggest chunk of the cost and you can probably replicate results of models like Llama 3 very cheaply. It's the cost of experiments, researchers, data collection which brings overall cost 1 or 2 order of magnitude higher.

What's your source for any of that? I think the $6 million thing was identified as a lie they felt was necessary because of GPU export laws.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#23

Post by kelipso »

lazide wrote: post #11
michelpp wrote: post #7

Earlier quoted context omitted.

Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind…

Memristors have been tried for literally decades. If the posters other guesses pay out the same rate, this will likely play out never.

There was a bit of noise regarding spiking neural networks a few years ago but now I am not seeing it so often anymore.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#24

Post by lazide »

ilaksh wrote: post #16
lazide wrote: post #11

Earlier quoted context omitted.

Memristors have been tried for literally decades. If the posters other guesses pay out the same rate, this will likely play out never.

Other technologies tried for decades before becoming huge: Neural-network AI; Electric cars; mRNA vaccines; Solar photovoltaics; LED lighting

Ho boy, should we start listing the 10x number of things that went in the wastebasket too?

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#25

Post by thomassmith65 »

ilaksh wrote: post #21
thomassmith65 wrote: post #17

Perhaps one of these days a random compsci undergrad will come up a DeepSeek-calibre optimization. Just imagine his or her 'ChatGPT with 10,000x fewer propagations' Reddit post appearing on a Monday... ...and $3 trillion of Nvidia stock going down the drain by Friday.

DeepSeek came up with several significant optimizations, not just one. And master's students do contribute to leading edge research all the time. There have really been many significant innovations in hardware, model architecture, and software, allowing companies to keep up with soaring demand and expectations. But that's always how it's been in high technology. You only really hear about the biggest shifts, but the…

True, but I chose the words 'ChatGPT' and 'optimization' for brevity. There are many more eyes on machine learning since ChatGPT came along. There could be simpler techniques yet to discover. What boggles the mind is the $4 trillion parked in Nvidia stock, and wasted if more efficient code lessens the need for expensive GPUs.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#26

Post by braooo »

stego-tech wrote: post #18
BoredPositron wrote: post #9

We haven't seen a proper npu and we are in the launch of the first consumer grade unified architectures by Nvidia and AMD. The battle of homebrew AI hasn't even started yet.

Hell, we haven’t even seen actual AI yet. This is all just brute-forcing likely patterns of tokens based on a corpus of existing material, not anything brand new or particularly novel. Who would’ve guessed that giving CompSci and Mathematics researchers billions of dollars in funding and millions of GPUs in parallel without the usual constraints of government research would produce the most expensive brute-force algo…

Yeh. We're still barely beyond the first few pixels that make up the bottom tail of the S-curve for autonomous type AI everyone imagines

Energy models and other substrates are going to be key, and it has nothing to do with text at all as human intelligence existed before language. It's Newspeak to run a chat bot on what is obviously a computer and call it an intelligence like a human. 1984 like dystopia crap.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#27

Post by cma »

hnuser123456 wrote: post #12
ilaksh wrote: post #4

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot o…

>memory-centric compute This already exists: https://www.cerebras.ai/chip They claim 44 GB of SRAM at 21 PB/s.

They use separate memory servers, networked memory adjacent adjacent compute with small amounts of fast local memory.

Waferscale severely limits bandwidth once you go beyond SRAM, because with far less chip perimeter per unit area there is less place to hook up IO.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#28

Post by ToValueFunfetti »

lazide wrote: post #24
ilaksh wrote: post #16

Earlier quoted context omitted.

Other technologies tried for decades before becoming huge: Neural-network AI; Electric cars; mRNA vaccines; Solar photovoltaics; LED lighting

Ho boy, should we start listing the 10x number of things that went in the wastebasket too?

If I only have to try 11 things for one of them to be LED lights or electric cars, I'd better get trying. Sure, I might have to empty a wastebasket at some point, but I'll just pay someone for that.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#29

Post by YetAnotherNick »

ilaksh wrote: post #22
YetAnotherNick wrote: post #14

Deepseek main run costed $6M. qwen3-30b-a3b probably would cost few $100Ks, which is ranked 13th. GPU cost of the final model training isn't the biggest chunk of the cost and you can probably replicate results of models like Llama 3 very cheaply. It's the cost of experiments, researchers, data collection which brings overall cost 1 or 2 order of magnitude higher.

What's your source for any of that? I think the $6 million thing was identified as a lie they felt was necessary because of GPU export laws.

It wasn't a lie, it was a misrepresentation of the total cost. It's not hard to calculate the cost of the training though. It takes 6 * active parameters * tokens flops[1]. To get number of seconds you can divide by Flops/s * MFU, where MFU is around 45% for H100 for large enough models[2].

[1]: https://arxiv.org/abs/2001.08361

[2]: https://github.com/facebookresearch/lingua

Post reply on HN