Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Meta's GenAI infrastructure

engineering.fb.com

Post reply on HN
Print view View original

141–150 of 314 posts

Re: Building Meta's GenAI infrastructure

#141

Post by karmasimida »

virtuallynathan wrote: post #126
lifeisstillgood wrote: post #60

Earlier quoted context omitted.

FB does not have the flywheel of running data centres - all three of those mentioned run hyper scale datacentres that they can then juice by “investing” billions in AI companies who then turn around and put those billions as revenue in the investors OpenAI takes money from MSFT and buys Azure services Anthropic takes Amazon money and buys AWS services (as do many robotics etc) I am fairly sure it’s not illegal but it…

Facebook has more datacenter space and power than Amazon, Google, and Microsoft -- possibly more than Amazon and Microsoft combined...

I don't think so, AWS hasn't disclosed this numbers, like datacenter spaces occupied, so how do you know.

Re: Building Meta's GenAI infrastructure

#143

Post by loeg »

latchkey wrote: post #33

> we have successfully used both RoCE and InfiniBand clusters for large, GenAI workloads (including our ongoing training of Llama 3 on our RoCE cluster) without any network bottlenecks. Interesting dig on IB. RoCE is the right solution since it is open standards and more importantly, available without a 52+ week lead time.

Yeah, and RoCE isn't single vendor. I'm not sure IB scales to the relevant cluster sizes, either.

Re: Building Meta's GenAI infrastructure

#144

Post by andy99 »

jabl wrote: post #120
ipsum2 wrote: post #118

Earlier quoted context omitted.

You're still bounded by memory bandwidth, so adding multiples to FLOPs is not going to give you a good representation of overall speedup.

Well, those smaller floats require less BW to transfer back and forth as well. Perhaps not a reduction linear in the size of the float, as maybe smaller floats require more iterations and/or more nodes in the model graph to get an equivalent result. But rest assured there's an improvement, it's not like people would be doing it if there wasn't any benefit!

The impact on bandwidth is the main reason smaller is better I belive, certainly when it's the bottleneck. I'm only really familiar with CPU but with say FP16 you might convert back to FP32 when you're doing the actual multiplication (so conversion plus multiplication is actually slower) but because you're moving half the data in and off you still get a huge speedup.

Re: Building Meta's GenAI infrastructure

#145

Post by jvanderbot »

KaiserPro wrote: post #128
jvanderbot wrote: post #110

So, I'd love to work on optimizing pipelines like this. How does one "get into" it? It seems a ML scientist with some C/C++ and infra knowledge just dips down into the system when required? Or is it CUDA/SIMD experts who move "up" into ML?

A lot of the optimisation at this level is getting data into the right place at the right time, without killing the network. Its also a group effort to provide simple to use primitives that "normal" ML people can use, even if they've never used hyper scale clusters before. So you need a good scheduler, that understand dependencies (no, the k8s scheduler(s) are shit for this, plus it wont scale past 1k nodes without e…

Ok, but back to my main question, how do I get into this?

Re: Building Meta's GenAI infrastructure

#146

Post by munificent »

crmd wrote: post #127
echelon wrote: post #87

Earlier quoted context omitted.

> If you want anything good, yes. If you just want something ... You don't even need AI for that. https://en.wikipedia.org/wiki/YouTube_poop https://en.wikipedia.org/wiki/Skibidi_Toilet The idea that AI isn't going to be used as a creative tool too and that it won't lead to more and better art is a defeatist, Luddite attitude. Similarly shaped people thought that digital cameras would ruin cinema and photography. > S…

>You'll be able to follow that steampunk vampire creator that's been missing from the world until now. Every long tail interest will be catered to. Even the most obscure and wild tastes, ideas, and designs. Stuff that would never get studio funding. Your optimism reminds me of the optimism I had around the early internet. Power to the people, long tail, rise of the creative class, the fall of gatekeeping corporations…

The big difference is that back then, anyone with a consumer-level computer in their bedroom could turn it into a server and be a first-class citizen on the Internet.

With generative AI, models will be controlled by a handful of giant corporations who have the enormous corpuses (of dubious provenance) and compute ability to train them.

So it will be like last time, but even worse.

Re: Building Meta's GenAI infrastructure

#147

Post by echelon »

munificent wrote: post #146
crmd wrote: post #127

Earlier quoted context omitted.

>You'll be able to follow that steampunk vampire creator that's been missing from the world until now. Every long tail interest will be catered to. Even the most obscure and wild tastes, ideas, and designs. Stuff that would never get studio funding. Your optimism reminds me of the optimism I had around the early internet. Power to the people, long tail, rise of the creative class, the fall of gatekeeping corporations…

The big difference is that back then, anyone with a consumer-level computer in their bedroom could turn it into a server and be a first-class citizen on the Internet. With generative AI, models will be controlled by a handful of giant corporations who have the enormous corpuses (of dubious provenance) and compute ability to train them. So it will be like last time, but even worse.

You can run ComfyUI and AnimateDiff on your PC. If you haven't checked them out, please do.

And there are other angles to consider. Apple, for one, is expressly interested in not becoming a thin client to cloud AI. They're baking a lot of inference power into their chips. If the creative class don't need their devices, that doesn't bode well for them...

Re: Building Meta's GenAI infrastructure

#148

Post by pgwhalen »

virtuallynathan wrote: post #126
lifeisstillgood wrote: post #60

Earlier quoted context omitted.

FB does not have the flywheel of running data centres - all three of those mentioned run hyper scale datacentres that they can then juice by “investing” billions in AI companies who then turn around and put those billions as revenue in the investors OpenAI takes money from MSFT and buys Azure services Anthropic takes Amazon money and buys AWS services (as do many robotics etc) I am fairly sure it’s not illegal but it…

Facebook has more datacenter space and power than Amazon, Google, and Microsoft -- possibly more than Amazon and Microsoft combined...

I have zero evidence, but this seems extremely unlikely. Do you have more than zero evidence?

Re: Building Meta's GenAI infrastructure

#149

Post by jedberg »

Legend2440 wrote: post #36
CuriouslyC wrote: post #18

Earlier quoted context omitted.

In pretty much every interview, Yann has talked about how important that AI infrastructure is open and distributed for the good of humanity, and how he wouldn't work for a company that wasn't open. Since Mark doesn't have an AI product to cannibalize, it's in his interest to devalue the AI products of others ("salting the earth").

I don't see how they're devaluing other people's AI products.

It's called commoditize the compliment.

If they make AI models free to use it makes OpenAI nearly valueless, which means that they can't survive and then sell Meta's competitors a better GenAI product than Meta can make themselves.

So basically since they don't make money directly on GenAI, it makes sense for them to release it for free so no one else can have something better, so they don't have to compete on GenAI abilities with their competitors.

Re: Building Meta's GenAI infrastructure

#150

Post by dekhn »

crakenzak wrote: post #40
aeyes wrote: post #20

Earlier quoted context omitted.

Isn't Google trying to do this with their TPUs?

I still, for the life of me, can't understand why Google doesn't just start selling their TPUs to everyone. Nvidia wouldn't be anywhere near their size if they only made H100s available through their DGX cloud, which is what Google is doing only making TPUs available through Google Cloud. Good hardware, good software support, and market is starving for performant competitors to the H100s (and soon B100s). Would sell…

Do you mean, sell TPU hardware to other companies that would run it in their data centers? I can't imagine that would ever really work. The only reason TPUs work at Google is because they have huge teams across many different areas to keep them running (SRE, hardware repair, SWE, hardware infra) and it's coupled to the design of the data centers. To vend and externalize the software would require google to setup similar teams for external customers (well beyond what Google Cloud provides for TPUs today) just to eke out some margin of profit. Plus, there is a whole proprietary stack running under the hood that google wouldn't want to share with potential competitors.

Google used to sell a search appliance-in-a-box and eventually lost interest because hardware is so high-touch.

Post reply on HN