Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta trains large language models at scale

engineering.fb.com

Post reply on HN
Print view View original

131–140 of 213 posts

Re: How Meta trains large language models at scale

#131

Post by girvo »

throwaway_ab wrote: post #15
mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

> If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone.

While I do not actually think Google's chips are better or close to being better, I don't think this actually holds?

If the upside of is effectively unbounded, it would outweigh the short term benefit of selling them to others, I would think. At least for a company like Google.

Re: How Meta trains large language models at scale

#132

Post by TeMPOraL »

bluedino wrote: post #59
throwaway_ab wrote: post #15

Earlier quoted context omitted.

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?

So you're saying, H100s are the corporate equivalent of Raspberry Pis now? Bought to not miss out, then left to gather dust in a drawer?

Re: How Meta trains large language models at scale

#133

Post by TeMPOraL »

mike_d wrote: post #116
stygiansonic wrote: post #22

Earlier quoted context omitted.

Yeah, ops comment makes it seem like they are building racks of RTX 4090s, when this isn’t remotely true. Tensor Core performance is far different on the data center class devices vs consumer ones.

They are building racks of 4090s. Nobody can get H100s in any reasonable volume. Hell, Microsoft is renting GPUs from Oracle Cloud to get enough capacity to run Bing.

There are apparently some 400 of H100s sitting idle somewhere upthread. Yes, I'm having hard time imagining how's that possible too.

Re: How Meta trains large language models at scale

#134

Post by logicchains »

tmostak wrote: post #76
jeffbee wrote: post #38

Earlier quoted context omitted.

I don't see how you can evaluate better and worse for training without doing so on cost basis. If it costs less and eventually finishes then it's better.

This assumes that you can linearly scale up the number of TPUs to get equal performance to Nvidia cards for less cost. Like most things distributed, this is unlikely to be the case.

This is absolutely the case, TPUs scale very well: https://github.com/google/maxtext .

Re: How Meta trains large language models at scale

#135

Post by whiplash451 »

idkdotcom wrote: post #54

These seem classic challenges with running distributed systems loads that are not specific to training LLMs. Anyone of the super computers listed here https://en.wikipedia.org/wiki/TOP500 suffers from the same issues. Think about it. While the national labs use these systems to model serious stuff -such as climate or nuclear weapons- Meta uses them to train LLMs. What a joke, honestly!

A lot of serious things look like a toy or a joke at first.

Re: How Meta trains large language models at scale

#136

Post by checkyoursudo »

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

What would any company as "the long term AI winner" look like? What would it mean to be the winner in this context?

Re: How Meta trains large language models at scale

#137

Post by logicchains »

fnordpiglet wrote: post #117
sangnoir wrote: post #102

Earlier quoted context omitted.

Are you suggesting Meta or Google - who stand to save billions - won't be able to get top performance from their custom chips because their tooling/hardware won't support CUDA?

No. I’m suggesting they won’t because IP like the mellanox treasure chest they acquired is ridiculously difficult to develop and Nvidia has aggressively exploited it, along with their other already advanced IP in the space of their -core business-. I understand, especially amongst googlers, there’s a belief there are no others smarter than a googler. But it’s simply not the case. Nvidia is excellent at their core com…

>No. I’m suggesting they won’t because IP like the mellanox treasure chest they acquired is ridiculously difficult to develop and Nvidia has aggressively exploited it, along with their other already advanced IP in the space of their -core business-.

For training Llama3 Facebook set up two clusters, one using fancy InfiniBand and one just using RoCE over Arista cards: https://engineering.fb.com/2024/03/12/data-center-engineerin... . The latter ended up doing fine, suggesting that all that Mellanox stuff isn't necessary for large-scale training (apparently at a large enough scale ethernet scales better than InfiniBand).

Re: How Meta trains large language models at scale

#138

Post by radarsat1 »

Frustratingly little information. For example, I'm exceedingly curious how they deal with scheduling jobs on such a huge array of machines. The article:

> Efficient scheduling helps ensure that our resources are used optimally. This involves sophisticated algorithms that can allocate resources based on the needs of different jobs and dynamic scheduling to adapt to changing workloads.

Wow thanks for that, captain obvious. So how do you do it?

Re: How Meta trains large language models at scale

#139

Post by logicchains »

houseplant wrote: post #104
mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

after everything I've seen and the litigation coming out of europe, I really can't see AI lasting long after they're obligated to prove rights for the data they're training on. they can't get away with having scraped people's owned work forever. You can't steal things from workers and then undercut them by selling that hard work for pennies, and not expect everything to collapse. I mean, I know that the folks in char…

Eventually they're going to put vision LLMs in robotic bodies and they'll be able to learn just by listening and watching, just like humans, at which stage the idea that they're "stealing" just by viewing content will be seen as absurd.

Re: How Meta trains large language models at scale

#140

Post by logicchains »

xvector wrote: post #56

> So we decided to build both: two 24k clusters, one with RoCE and another with InfiniBand. Our intent was to build and learn from the operational experience. I love how they built two completely insane clusters just to learn. That's badass.

It's not just to learn; an RoCE ethernet cluster with Aristas is way cheaper to build and maintain than a fancy InfiniBand cluster with Mellanox/NVidia networking, so proving that the former is good enough at scale will eventually save Meta a huge amount of money. InfiniBand cards are much more expensive than ethernet because there's few vendors, that have a quasi-monopoloy, and because overall far fewer of them are produced so there's less economy of scale.
Post reply on HN