How much are they paying for H100's? If they are paying $10k: 350,000 NVIDIA H100 x $10k = $3.5b
They may have to pay a premium to secure ~¼ of the output; certainly unlikely to be that steep a discount.
Building Meta's GenAI infrastructure
41–50 of 314 posts
Re: Building Meta's GenAI infrastructure
#42Earlier quoted context omitted.
In pretty much every interview, Yann has talked about how important that AI infrastructure is open and distributed for the good of humanity, and how he wouldn't work for a company that wasn't open. Since Mark doesn't have an AI product to cannibalize, it's in his interest to devalue the AI products of others ("salting the earth").
I don't see how they're devaluing other people's AI products.
The Llama models have played a large part in fostering the development of the open source LLM ecosystem, and I expect Llama3 to put in performance > mistral medium and anthropic haiku while being fully open and able to be run on consumer hardware.
Re: Building Meta's GenAI infrastructure
#43Earlier quoted context omitted.
In pretty much every interview, Yann has talked about how important that AI infrastructure is open and distributed for the good of humanity, and how he wouldn't work for a company that wasn't open. Since Mark doesn't have an AI product to cannibalize, it's in his interest to devalue the AI products of others ("salting the earth").
I don't see how they're devaluing other people's AI products.
The angle is that by releasing cutting edge AI research to the public openly, the relative difference between open source models/tech and closed source tech shrinks.
Whether or not you think the "value" of AI products is proportional to their performance gap vs the next closest thing or not is up to you. Very interesting PG essay I read recently talks about the opposite of this (Superlinear returns) where if you're half as good as the next competitor, you don't get half the customers, you get 0.
Re: Building Meta's GenAI infrastructure
#44Earlier quoted context omitted.
Isn't Google trying to do this with their TPUs?
I still, for the life of me, can't understand why Google doesn't just start selling their TPUs to everyone. Nvidia wouldn't be anywhere near their size if they only made H100s available through their DGX cloud, which is what Google is doing only making TPUs available through Google Cloud. Good hardware, good software support, and market is starving for performant competitors to the H100s (and soon B100s). Would sell…
Maybe selling hardware to customers worldwide + support like Nvidia does is actually not trivial ?
Re: Building Meta's GenAI infrastructure
#45Earlier quoted context omitted.
What do you mean?
In pretty much every interview, Yann has talked about how important that AI infrastructure is open and distributed for the good of humanity, and how he wouldn't work for a company that wasn't open. Since Mark doesn't have an AI product to cannibalize, it's in his interest to devalue the AI products of others ("salting the earth").
is "salting the earth", in the biblical sense of destroying your enemy and their land to the point where not even plants grow again, a SV term used for companies that promote open source?
Re: Building Meta's GenAI infrastructure
#46I'd be great if they could invest in an alternative to nvidia -- then, in one fell swoop, destroy the moats of everyone in the industry.
A company moving away from Nvidia/CUDA while the field is developing so rapidly would result in that company falling behind. When (if) the rate of progress in the AI space slows, then perhaps the big players will have the breathing room to consider rethinking foundational components of their infrastructure. But even at that point, their massive investment in Nvidia will likely render this impractical. Nvidia decisive…
People said the same thing when tensorflow was all the rage and pytorch was a side project.
Granted, HW is much harder than SW, but I would not discount Meta's ability to displace NVIDIA entirely.
Re: Building Meta's GenAI infrastructure
#47Earlier quoted context omitted.
It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.
What does video not be in the future mean? In social media tiktok and reels are everywhere?
There are reports [1] that a bunch of companies like "College Humor" were convinced to switch to producing native video for facebook (instead of directing users to their own sites) on the basis of bullshit metrics from facebook, and had an extremely bad time as a result, with some companies going bankrupt.
Something like counting an autoplaying video that ran for 3 seconds as a 'view' IIRC
[1] https://twitter.com/adamconover/status/1183209875859333120
Re: Building Meta's GenAI infrastructure
#48> At Meta, we handle hundreds of trillions of AI model executions per day Such a large number, makes sense?
Sure. 100T/day * 1day/86400sec ~= 1B/sec. They're probably considering at least a few hundred candidates per impression, and every impression is going to go through _at least_ two models (relevance and pCTR/revenue), so you could get there just with online serving at 5Mqps, which is plausible. But they're also going to be doing a lot of stuff in batch - spam predictions, ad budget forecasts, etc - so that every candidate actually runs through four or five different models, and every actual impression could do more than that.
Re: Building Meta's GenAI infrastructure
#49Earlier quoted context omitted.
It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.
What does video not be in the future mean? In social media tiktok and reels are everywhere?
They are referring to Facebook/Meta’s 2015 “pivot to video”, speculating there may be a similar thing happening more recently with AI.
Re: Building Meta's GenAI infrastructure
#50How much are they paying for H100's? If they are paying $10k: 350,000 NVIDIA H100 x $10k = $3.5b
> $3.5b
Which is a fourth of what they spent in VR/AR in a year. And Gen AI is something they could easily get more revenue as it has now become proven technology, and Meta could possibly leapfrog others because of the data moat.