Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPU-rich labs have won: What's left for the rest of us is distillation

inference.net

Post reply on HN
Print view View original

31–40 of 53 posts

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#31

Post by manquer »

thekoma wrote: post #8
ilaksh wrote: post #4

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot o…

Even in that scenario, what would stop the likes of OpenAI to instead throw 50M+ a day to the new way of doing things and still outcompete smaller fry?

The fastest away to acquire the know-how to do for Big Co is to get the talent who have spent the years in building the new tech.

Poaching, acquihirng or acquisitions and the myriad modern forms we are seeing today have been the tools and will not change.

Owners and beneficiaries of the capital do not change, but that is an artifact of our economic system and is much larger a socio-economic discussion beyond the scope of innovation and research

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#32

Post by Mars008 »

stego-tech wrote: post #18
BoredPositron wrote: post #9

We haven't seen a proper npu and we are in the launch of the first consumer grade unified architectures by Nvidia and AMD. The battle of homebrew AI hasn't even started yet.

Hell, we haven’t even seen actual AI yet. This is all just brute-forcing likely patterns of tokens based on a corpus of existing material, not anything brand new or particularly novel. Who would’ve guessed that giving CompSci and Mathematics researchers billions of dollars in funding and millions of GPUs in parallel without the usual constraints of government research would produce the most expensive brute-force algo…

It's a necessary evolution step. Did you know our own ancestors had tails and grills. Do you feel ashamed?

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#33

Post by joshcartme »

Maybe I'm totally misreading this, but it seems like the post contradicts itself. At the beginning of the third paragraph:

> Impressively, open source models have been able to quickly catch up to big labs.

And then the beginning of the fourth:

> Open-source has been lagging behind proprietary models for years, but lately this gap has been widening.

Followed by a picture that is more or less inscrutable.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#34

Post by hyperbovine »

michelpp wrote: post #7
ilaksh wrote: post #4

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot o…

Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind…

> Sparse models for example show a lot of promise and more closely track real biological systems.

I think sparsity is a consequence of some other fundamental properties of brain function that we've yet to understand. Just sparsifying the models we've got is not going to lead anywhere, IMO. (For example it's estimated that current AI models are already within 1%-10% of a human brain in terms of "number of parameters" (https://www.beren.io/2022-08-06-The-scale-of-the-brain-vs-ma...).)

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#36

Post by roenxi »

joshcartme wrote: post #33

Maybe I'm totally misreading this, but it seems like the post contradicts itself. At the beginning of the third paragraph: > Impressively, open source models have been able to quickly catch up to big labs. And then the beginning of the fourth: > Open-source has been lagging behind proprietary models for years, but lately this gap has been widening. Followed by a picture that is more or less inscrutable.

> Followed by a picture that is more or less inscrutable.

Yeah. Just to make it explicit - that chart has Deepseek r1 at ... presumably an elo of 1418 and Gemini Pro at 1463. That is comparable to the gap between Magnus Carlsen and Fabiano Caruana [0]. I don't think it is reasonable to complain about that sort of performance gap in practice - it is a capable model. Looking at the spread of scores I don't immediately see why someone even needs to use something in the Top 10, presumably anything above 1363 would be good enough for business, research and personal use.

None of these models have even been around that long, Deepseek was only released in January. The rate of change is massive, I expect to have access to an open source model that is better than anything on this leaderboard next year some time.

[0] https://2700chess.com/

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#38

Post by npmipg »

joshcartme wrote: post #33

Maybe I'm totally misreading this, but it seems like the post contradicts itself. At the beginning of the third paragraph: > Impressively, open source models have been able to quickly catch up to big labs. And then the beginning of the fourth: > Open-source has been lagging behind proprietary models for years, but lately this gap has been widening. Followed by a picture that is more or less inscrutable.

Hey, I'm the author of the post.

The image has been fixed, and the point I'm making is that proprietary models are almost always ahead, and this gap is widening. OS models that are nearly at the same quality are usually distilled versions of proprietary models, or somehow get training data from them. Sometimes, after massive, expensive training runs models are open sourced anyway, and at some point that becomes unsustainable.

The difference between a top model and a model with a similar ELO might seem small, but the value of even a marginal increase in intelligence is extremely high--for example I only use the best coding model for coding, whatever the cost.

There's also lots of evidence that large labs are only getting started. In the past year, they have secured massive amounts of compute, which is still not utilized well. I expect lots of big training runs in the future, which will shift the gap further between OS and proprietary models.

The major problem for these companies is they spend hundreds of millions of dollars training a model, and then someone comes in the next day and distills something almost as good for far less money (still a VERY large sum of money.)

I don't know how this will be resolved long term.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#39

Post by jijijijij »

Mars008 wrote: post #32
stego-tech wrote: post #18

Earlier quoted context omitted.

Hell, we haven’t even seen actual AI yet. This is all just brute-forcing likely patterns of tokens based on a corpus of existing material, not anything brand new or particularly novel. Who would’ve guessed that giving CompSci and Mathematics researchers billions of dollars in funding and millions of GPUs in parallel without the usual constraints of government research would produce the most expensive brute-force algo…

It's a necessary evolution step. Did you know our own ancestors had tails and grills. Do you feel ashamed?

No, but maybe pandas should. You know, evolution is mostly dead ends, literally.

Re: GPU-rich labs have won: What's left for the rest of us is distillation

#40

Post by npmipg »

npmipg wrote: post #38
joshcartme wrote: post #33

Maybe I'm totally misreading this, but it seems like the post contradicts itself. At the beginning of the third paragraph: > Impressively, open source models have been able to quickly catch up to big labs. And then the beginning of the fourth: > Open-source has been lagging behind proprietary models for years, but lately this gap has been widening. Followed by a picture that is more or less inscrutable.

Hey, I'm the author of the post. The image has been fixed, and the point I'm making is that proprietary models are almost always ahead, and this gap is widening. OS models that are nearly at the same quality are usually distilled versions of proprietary models, or somehow get training data from them. Sometimes, after massive, expensive training runs models are open sourced anyway, and at some point that becomes unsus…

Note that distilling a general model is several orders of magnitude more expensive than distilling a task-specific model, which is what I'm trying to promote here. Smart general models make distilling great task specific models with no expert labelers way easier.
Post reply on HN