Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta trains large language models at scale

engineering.fb.com

Post reply on HN
Print view View original

111–120 of 213 posts

Re: How Meta trains large language models at scale

#111

Post by uluyol »

vineyardmike wrote: post #100
throwaway_ab wrote: post #15

Earlier quoted context omitted.

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

> why wouldn't Google be selling shovels They do sell them - but through their struggling cloud business. Either way, Nvidia's margin is google's opportunity to lower costs. > I can see Google's custom chips are 15x to 30x slower to train AI TPUs are designed for inference not training - they're betting that they can serve models to the world at a lower cost structure than their competition. The compute required for…

Google most certainly uses TPUs for training.

Re: How Meta trains large language models at scale

#112

Post by smueller1234 »

refulgentis wrote: post #110
vineyardmike wrote: post #100

Earlier quoted context omitted.

> why wouldn't Google be selling shovels They do sell them - but through their struggling cloud business. Either way, Nvidia's margin is google's opportunity to lower costs. > I can see Google's custom chips are 15x to 30x slower to train AI TPUs are designed for inference not training - they're betting that they can serve models to the world at a lower cost structure than their competition. The compute required for…

This breaks my brain, because I know Google trains it models on TPUs and they're seen as faster, and if they're better at inference, and can train, then why is Nvidia in a unique position? My understanding was always it's as simple as it required esoteric tooling

Multiple types of TPUs.

(I work for Google, but the above is public information.)

Re: How Meta trains large language models at scale

#113

Post by mike_d »

matt-p wrote: post #26
mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

Can't agree. This is like saying $popularApp will fail because they buy expensive hosting at AWS. Rubbish they will fail because the product didn't fit the market, if they're successful they'll have money to buy servers and colo then drive down cost. If they succeed it will be in large part due to the fact they spent thier capital and more importantly time on code/engineers rather than servers. Right now companies ar…

> This is like saying $popularApp will fail because they buy expensive hosting at AWS.

For any given mobile app startup, AWS is effectively infinite. The more money you throw at it the more doodads you get back. Nvidia's supply chain is not infinite and is the bottle neck for all the non-Google players to fight over.

Re: How Meta trains large language models at scale

#114

Post by iamflimflam1 »

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

Don’t forget Apple’s Private Compute Cloud - built on top of Apple Silicon.

Re: How Meta trains large language models at scale

#116

Post by mike_d »

stygiansonic wrote: post #22
zdyn5 wrote: post #6

Earlier quoted context omitted.

H100s are far from consumer video cards

Yeah, ops comment makes it seem like they are building racks of RTX 4090s, when this isn’t remotely true. Tensor Core performance is far different on the data center class devices vs consumer ones.

They are building racks of 4090s. Nobody can get H100s in any reasonable volume.

Hell, Microsoft is renting GPUs from Oracle Cloud to get enough capacity to run Bing.

Re: How Meta trains large language models at scale

#117

Post by fnordpiglet »

sangnoir wrote: post #102
fnordpiglet wrote: post #47

Earlier quoted context omitted.

The problem isn’t just developing your own processor. Nvidia has a huge stack of pretty cutting edge technology including a huge stack from mellanox, an enormous OSS tool chain around CUDA, etc, that people seeking to make comparable products have to overcome.

Are you suggesting Meta or Google - who stand to save billions - won't be able to get top performance from their custom chips because their tooling/hardware won't support CUDA?

No. I’m suggesting they won’t because IP like the mellanox treasure chest they acquired is ridiculously difficult to develop and Nvidia has aggressively exploited it, along with their other already advanced IP in the space of their -core business-.

I understand, especially amongst googlers, there’s a belief there are no others smarter than a googler. But it’s simply not the case. Nvidia is excellent at their core competencies and business, which is making absurdly parallel compute platforms with absurdly powerful interconnects. I’m saying google or meta won’t beat Nvidia at hardware. I’d also point to the fact Nvidias ability to raise capital is the best on earth now, so even money isn’t a barrier.

The advantage CUDA gives is in the tool chains, libraries, research, and all that that tens of thousands of people are contributing to as part of their jobs, research, and hobbies. This is almost -more valuable- than getting top performance. Getting top techniques, top software, top everything by having everyone everywhere working to make the ecosystem of your stuff is invaluable. Google won’t have that. They will just have the hubris of googlers who believe they’re smarter.

I would also note that at this phase of a cycle in tech trying to save billions takes your eye off the prize. Cost optimization comes much later after the market has been fully explored and directions are clean and diminishing returns on R&D kick in. Any company that doesn’t recognize that is run by CPA and deserves the ignominy they’ll face.

Re: How Meta trains large language models at scale

#118

Post by towawy »

iamflimflam1 wrote: post #114
mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

Don’t forget Apple’s Private Compute Cloud - built on top of Apple Silicon.

Models trained on Google TPUs according to Reuters [0]. Does anyone know the "technical document" the news article references?

[0] https://www.reuters.com/technology/artificial-intelligence/h...

Re: How Meta trains large language models at scale

#120

Post by TeMPOraL »

yosito wrote: post #67

I wish that instead of just training another stupid LLM, Meta would use it to improve their search and help me find the content I'm actually interested in.

Their revenue depends on it being hard (but not impossible) for you to find the content you're actually interested in. Would be nice if it didn't, but in this reality, money on the Internet is made by wasting users' lives. That is what attention economy is about.
Post reply on HN