Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta trains large language models at scale

engineering.fb.com

Post reply on HN
Print view View original

101–110 of 213 posts

Re: How Meta trains large language models at scale

#101

Post by jedberg »

jonathanlei wrote: post #94
bluedino wrote: post #59

Earlier quoted context omitted.

We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?

Hello! If you're interested in monetizing those GPUs, I'd be happy to rent them (all 400!) and offer those to customers of the cloud I work at :) jonathan [at] tensordock.com

If you want 512 H100s connected with infiniband: https://lambdalabs.com/service/gpu-cloud/1-click-clusters

Re: How Meta trains large language models at scale

#102

Post by sangnoir »

fnordpiglet wrote: post #47
moneywoes wrote: post #18

Earlier quoted context omitted.

Is no one else working on custom silicon?

The problem isn’t just developing your own processor. Nvidia has a huge stack of pretty cutting edge technology including a huge stack from mellanox, an enormous OSS tool chain around CUDA, etc, that people seeking to make comparable products have to overcome.

Are you suggesting Meta or Google - who stand to save billions - won't be able to get top performance from their custom chips because their tooling/hardware won't support CUDA?

Re: How Meta trains large language models at scale

#104

Post by houseplant »

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

after everything I've seen and the litigation coming out of europe, I really can't see AI lasting long after they're obligated to prove rights for the data they're training on.

they can't get away with having scraped people's owned work forever. You can't steal things from workers and then undercut them by selling that hard work for pennies, and not expect everything to collapse. I mean, I know that the folks in charge of this aren't really known for their foresight, especially when stock numbers and venture capital are the entire point, but... surely I hope people can recognize that this can't go on unimpeded.

Re: How Meta trains large language models at scale

#105

Post by throwup238 »

cachvico wrote: post #79
samspenc wrote: post #52

OK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line GPUs falling off: In this case, GPUs are not detected by the host on PCIe.

Brings a whole new meaning to bus factor

I’ve never met a GPU that could survive getting hit by a bus.

Re: How Meta trains large language models at scale

#106

Post by [deleted] »

loeg wrote: post #16
mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

The only thing that can stop Google is Google. Somehow every bet that isn't Search doesn't pan out. And inexplicably, they're working hard to kill Search now. As a shareholder, I hope they succeed. But I am more pessimistic about it than you.

[deleted]

Re: How Meta trains large language models at scale

#107

Post by Der_Einzige »

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

Until the lion share of AI projects support that custom silicon, I will continue to bet on anyone buying Nvidia GPUs.

Re: How Meta trains large language models at scale

#108

Post by Der_Einzige »

bluedino wrote: post #59
throwaway_ab wrote: post #15

Earlier quoted context omitted.

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?

I know of many important projects that need GPUs right now and aren’t getting any. You could help motivate the ponydiffusion folks to actually try finetuning of SD3!

Re: How Meta trains large language models at scale

#109

Post by ChuckMcM »

samspenc wrote: post #52

OK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line GPUs falling off: In this case, GPUs are not detected by the host on PCIe.

The bits on the bus go round and round!

There is a lot of interesting yet unpublished work on 'data center' scale compute complexes. It was a rabbit hole I fell into several times while at Google.

Re: How Meta trains large language models at scale

#110

Post by refulgentis »

vineyardmike wrote: post #100
throwaway_ab wrote: post #15

Earlier quoted context omitted.

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

> why wouldn't Google be selling shovels They do sell them - but through their struggling cloud business. Either way, Nvidia's margin is google's opportunity to lower costs. > I can see Google's custom chips are 15x to 30x slower to train AI TPUs are designed for inference not training - they're betting that they can serve models to the world at a lower cost structure than their competition. The compute required for…

This breaks my brain, because I know Google trains it models on TPUs and they're seen as faster, and if they're better at inference, and can train, then why is Nvidia in a unique position? My understanding was always it's as simple as it required esoteric tooling
Post reply on HN