Metas backing itself into a corner with its admirable commitment to open source. Unfortunately, at some point when they decide to monetize their billions spent and try to release a closed source model, the level of vitriol they will deal with will be an order of magnitude above what even OpenAI is experiencing. I don’t think they realize that!
Building Meta's GenAI infrastructure
231–240 of 314 posts
Re: Building Meta's GenAI infrastructure
#232Metas backing itself into a corner with its admirable commitment to open source. Unfortunately, at some point when they decide to monetize their billions spent and try to release a closed source model, the level of vitriol they will deal with will be an order of magnitude above what even OpenAI is experiencing. I don’t think they realize that!
The general public doesn’t care. Only developers.
Re: Building Meta's GenAI infrastructure
#233Metas backing itself into a corner with its admirable commitment to open source. Unfortunately, at some point when they decide to monetize their billions spent and try to release a closed source model, the level of vitriol they will deal with will be an order of magnitude above what even OpenAI is experiencing. I don’t think they realize that!
No
Meta's commitment to Open Source is well under calculation.
OCP is a way to rally lower-tier vendors to form a semi-alliance to keep up with super-gorilla like AWS & Google.
LLaMA has already gained much more than its cost (look at the stock price, and the open source ecosystem built surrounding LLaMA, and Google's open source Gemma models which is a proof of Meta's success).
IMHO, Meta's Open Source strategy already covered at least 5 years in prospect. That's enough to finesse a 180 degree turn around if necessary (i.e., from open source to close source)
Re: Building Meta's GenAI infrastructure
#234Re: Building Meta's GenAI infrastructure
#235Earlier quoted context omitted.
What does that have to do with Gen AI
GenAI infra is the same as regular AI infra. They used GenAI in the title because it's a buzzword.
Not really. Ranking and recommendation models require different infrastructure than LLMs. The models are generally smaller and require more data processing before training.
Re: Building Meta's GenAI infrastructure
#236float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?
You're still bounded by memory bandwidth, so adding multiples to FLOPs is not going to give you a good representation of overall speedup.
I'm not sure exactly on how NVIDIA calculates FLOPs, but I do know for Intel's FLOPs, it's calculated from how many FMA units, how many loads can be done in tandem, and what the throughput is. And ye fp8 requires 2x less space. Sparse 2:4 might be less pronounced, since the matrix first needs to be constructed on the fly, and there is like a small matrix of indicator values.
Re: Building Meta's GenAI infrastructure
#237Earlier quoted context omitted.
Well, those smaller floats require less BW to transfer back and forth as well. Perhaps not a reduction linear in the size of the float, as maybe smaller floats require more iterations and/or more nodes in the model graph to get an equivalent result. But rest assured there's an improvement, it's not like people would be doing it if there wasn't any benefit!
The impact on bandwidth is the main reason smaller is better I belive, certainly when it's the bottleneck. I'm only really familiar with CPU but with say FP16 you might convert back to FP32 when you're doing the actual multiplication (so conversion plus multiplication is actually slower) but because you're moving half the data in and off you still get a huge speedup.
I can't remember some research paper somewhere even if you do float32 multiplications, but keep the data in bfloat16 by just simply truncating the lower mantissa bits, and doing packing, you still get speedups, since matrix multiplication is bound both by compute and cache access. If you can optimize on the cache side of things, speedups are definitely there.
Re: Building Meta's GenAI infrastructure
#238Earlier quoted context omitted.
Nope. Moreover, simulating it even with AVX-512 is quite an experience. Been postponing it for 2 years now... But first of all, you need to choose the version of float8 you want to implement, as the standards differ between GPU vendors.
We use it in gemma.cpp [1]. This hybrid of E5M2 and E4M3 decodes to bf16 in ~14 instructions, so we can do that on the fly during dot products. [1]: github.com/google/gemma.cpp
Congratulations on gemma.cpp!!
Re: Building Meta's GenAI infrastructure
#239Re: Building Meta's GenAI infrastructure
#240Having lived through the dot-com era, I find the AI-era slightly dispiriting because of the sheer capital cost of training models. At the start of the dot-com era, anyone could spin up an e-commerce site with relatively little infrastructure costs. Now, it seems, only the hyper-scale companies can build these AI models. Meta, Google, Microsoft, Open-AI, etc.
Not everything has to be AI. You can run a small business infra for MUCH less than you did back then, especially if you adjust for inflation (!). Training AI models costs a fortune, but so far it's been just front-loading costs in hopes of a windfall. We'll see what actually happens.
Front loading costs to eventually extract rents on usage with one hell of a capital wall protecting the assets.
Its easier to spin up a business for sure -- also easier to unwind it - there not as sticky as they used to be.