350k H100 cards, around ten billion dollars just for the GPUs. Less if Nvidia gives a volume discount, which I imagine they do not.
Building Meta's GenAI infrastructure
21–30 of 314 posts
Re: Building Meta's GenAI infrastructure
#22I'd be great if they could invest in an alternative to nvidia -- then, in one fell swoop, destroy the moats of everyone in the industry.
Facebook very specifically bought and customized Intel SKUs tailored for AI workloads for some time.
Re: Building Meta's GenAI infrastructure
#23The article doesn't mention MTIA, meta's custom ASIC for training & inference acceleration.
https://ai.meta.com/blog/meta-training-inference-accelerator...
I wonder if they will use it in RSC.
Re: Building Meta's GenAI infrastructure
#24Searched H100 and an Amazon link popped up. Good reviews.
https://www.amazon.com/Tesla-NVIDIA-Learning-Compute-Graphic...
Re: Building Meta's GenAI infrastructure
#25350k H100 cards, around ten billion dollars just for the GPUs. Less if Nvidia gives a volume discount, which I imagine they do not.
It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.
Re: Building Meta's GenAI infrastructure
#26> At Meta, we handle hundreds of trillions of AI model executions per day
Such a large number, makes sense?
Re: Building Meta's GenAI infrastructure
#27float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?
Re: Building Meta's GenAI infrastructure
#28350k H100 cards, around ten billion dollars just for the GPUs. Less if Nvidia gives a volume discount, which I imagine they do not.
It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.
[deleted]
Re: Building Meta's GenAI infrastructure
#29> Meta’s long-term vision is to build artificial general intelligence (AGI)
Re: Building Meta's GenAI infrastructure
#30> At Meta, we handle hundreds of trillions of AI model executions per day Such a large number, makes sense?
Perhaps there's some combinatorics where every time an ad or post is displayed to the user, it runs through some hundreds/thousands of candidates and computes their relevance.