Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Meta's GenAI infrastructure

engineering.fb.com

Post reply on HN
Print view View original

31–40 of 314 posts

Re: Building Meta's GenAI infrastructure

#31

Post by motoxpro »

renegade-otter wrote: post #25
hendersoon wrote: post #21

350k H100 cards, around ten billion dollars just for the GPUs. Less if Nvidia gives a volume discount, which I imagine they do not.

It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.

It already paid off. When the world moved from determinisic to probablistic ad modeling. That's why their numbers are so good right now compared to every other advertiser

Re: Building Meta's GenAI infrastructure

#32

Post by echelon »

renegade-otter wrote: post #25
hendersoon wrote: post #21

350k H100 cards, around ten billion dollars just for the GPUs. Less if Nvidia gives a volume discount, which I imagine they do not.

It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.

As a practitioner in the field, I can assure you this is not a boondoggle.

Those GPUs are going to subsume the entire music, film, and gaming industries. And that's just to start.

Re: Building Meta's GenAI infrastructure

#33

Post by latchkey »

> we have successfully used both RoCE and InfiniBand clusters for large, GenAI workloads (including our ongoing training of Llama 3 on our RoCE cluster) without any network bottlenecks.

Interesting dig on IB. RoCE is the right solution since it is open standards and more importantly, available without a 52+ week lead time.

Re: Building Meta's GenAI infrastructure

#34

Post by tayo42 »

renegade-otter wrote: post #25
hendersoon wrote: post #21

350k H100 cards, around ten billion dollars just for the GPUs. Less if Nvidia gives a volume discount, which I imagine they do not.

It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.

What does video not be in the future mean? In social media tiktok and reels are everywhere?

Re: Building Meta's GenAI infrastructure

#36

Post by Legend2440 »

CuriouslyC wrote: post #18
bananabrick wrote: post #9

Earlier quoted context omitted.

What do you mean?

In pretty much every interview, Yann has talked about how important that AI infrastructure is open and distributed for the good of humanity, and how he wouldn't work for a company that wasn't open. Since Mark doesn't have an AI product to cannibalize, it's in his interest to devalue the AI products of others ("salting the earth").

I don't see how they're devaluing other people's AI products.

Re: Building Meta's GenAI infrastructure

#37

Post by Legend2440 »

baby wrote: post #14
choppaface wrote: post #13

Total cluster they say will reach 350k H100, which at $30k street price is about $10b. In contrast, Microsoft is spending over $10b per quarter capex on cloud. That makes Zuck look conservative after his big loss on metaverse. https://www.datacenterdynamics.com/en/news/q3-2023-cloud-res...

What loss lol. Stop the fud

Has literally anyone spent money on the metaverse? Maybe it'll still take off in the future, but it's a $40b loss so far.

Re: Building Meta's GenAI infrastructure

#38

Post by mjburgess »

math_dandy wrote: post #12
mjburgess wrote: post #7

I'd be great if they could invest in an alternative to nvidia -- then, in one fell swoop, destroy the moats of everyone in the industry.

A company moving away from Nvidia/CUDA while the field is developing so rapidly would result in that company falling behind. When (if) the rate of progress in the AI space slows, then perhaps the big players will have the breathing room to consider rethinking foundational components of their infrastructure. But even at that point, their massive investment in Nvidia will likely render this impractical. Nvidia decisive…

I'm more concerned to avoid nvidia (et al.) market domination, than chasing the top-edge of the genAI benefits sigmoid. This will prevent much broad-based innovation.

Re: Building Meta's GenAI infrastructure

#39

Post by GamerAlias »

danielhanchen wrote: post #27

float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?

I was thinking why is this one guy on HN so deeply interested and discussing technical details from a minor remark. Then I clocked the name. Great work on Gemma bugs

Re: Building Meta's GenAI infrastructure

#40

Post by crakenzak »

aeyes wrote: post #20
mjburgess wrote: post #7

I'd be great if they could invest in an alternative to nvidia -- then, in one fell swoop, destroy the moats of everyone in the industry.

Isn't Google trying to do this with their TPUs?

I still, for the life of me, can't understand why Google doesn't just start selling their TPUs to everyone. Nvidia wouldn't be anywhere near their size if they only made H100s available through their DGX cloud, which is what Google is doing only making TPUs available through Google Cloud.

Good hardware, good software support, and market is starving for performant competitors to the H100s (and soon B100s). Would sell like hotcakes.

Post reply on HN