[dead]
Building Meta's GenAI infrastructure
221–230 of 314 posts
Re: Building Meta's GenAI infrastructure
#222Earlier quoted context omitted.
Unless you've worked at Amazon, Microsoft, Google, and Facebook, or a whole bunch of datacenter providers, I'm not sure how you could make that claim. They don't really share that information freely, even in their stock reports. Heck I worked at Amazon and even then I couldn't tell you the total datacenter space, they don't even share it internally.
You can just map them all... I have. I also worked at AWS :)
Mapping as in.. drawing the outlines of buildings and computing the square footage yourself?
Re: Building Meta's GenAI infrastructure
#223All this compute and my Instagram Reels feed still isn't as good as my TikTok feed
Re: Building Meta's GenAI infrastructure
#224Earlier quoted context omitted.
The angle is that by releasing cutting edge AI research to the public openly, the relative difference between open source models/tech and closed source tech shrinks. Whether or not you think the "value" of AI products is proportional to their performance gap vs the next closest thing or not is up to you. Very interesting PG essay I read recently talks about the opposite of this (Superlinear returns) where if you're h…
New Linux versions don't "salt the earth" for Windows.
Windows should always provide enough additional value that makes up for what they are asking as money - compared to the free option. That is the point. If you had no other viable options, then they could do whatever they like. Now they have a baseline to compete with and it is very hard to compete with free.
Re: Building Meta's GenAI infrastructure
#225float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?
care to explain why attention has precision issues with fp8?
Oh so float8's L2 Norm from float32 is around I think 1e-4, whilst float16 is 1e-6. Sadly attention is quite sensitive. There are some hybrid methods which just before the attention kernel which is done in fp8, upcasts the Q and K from the RoPE kernel to become float16, then also leaves V to be in float8. Everything is done in fp8 on the fly, and the output is fp8. This makes errors go to 1e-6.
Re: Building Meta's GenAI infrastructure
#226So, I'd love to work on optimizing pipelines like this. How does one "get into" it? It seems a ML scientist with some C/C++ and infra knowledge just dips down into the system when required? Or is it CUDA/SIMD experts who move "up" into ML?
I work on PyTorch Compilers at Meta, and I think folks enter ML Systems from all directions :)
Some folks start with more familiarity in ML research and dip down as far as they need.
Other folks come from a traditional distributed systems/compilers/HPC background, and apply those skills to ML systems.
Re: Building Meta's GenAI infrastructure
#227Re: Building Meta's GenAI infrastructure
#228Earlier quoted context omitted.
Is there float8 support in any common CPU intrinsics? It sounds interesting but curious what will be the impact if any on CPU inference.
Nope. Moreover, simulating it even with AVX-512 is quite an experience. Been postponing it for 2 years now... But first of all, you need to choose the version of float8 you want to implement, as the standards differ between GPU vendors.
We use it in gemma.cpp [1]. This hybrid of E5M2 and E4M3 decodes to bf16 in ~14 instructions, so we can do that on the fly during dot products.
[1]: github.com/google/gemma.cpp
Re: Building Meta's GenAI infrastructure
#229All this compute and my Instagram Reels feed still isn't as good as my TikTok feed
What does that have to do with Gen AI
Re: Building Meta's GenAI infrastructure
#230Having lived through the dot-com era, I find the AI-era slightly dispiriting because of the sheer capital cost of training models. At the start of the dot-com era, anyone could spin up an e-commerce site with relatively little infrastructure costs. Now, it seems, only the hyper-scale companies can build these AI models. Meta, Google, Microsoft, Open-AI, etc.
We will probably get there, it's just going to take time for hardware supply chains to catch up. I feel it's more comparable to mainframe eras - it took time for general purpose computing to become commoditised.