Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta MTIA v2 – Meta Training and Inference Accelerator

ai.meta.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#51

Post by giantrobot »

noiseinvacuum wrote: post #45
Aurornis wrote: post #22

Earlier quoted context omitted.

> Yeah, it may fit their current workload perfectly, but it doesn't seem very future proof It’s custom silicon designed for a specific, known workload. It’s not designed to be a general purpose part or to be future proofed for unknown future applications. When a new application comes along with new requirements, the teams will use their experience to create a new chip targeting that new application. That’s the great…

Right and they have a LOT of GPUs from Nvidia for handle the unknown. Custom silicon for custom workloads seems like a good strategy specially considering the capabilities that the team will develop along the way.

Offloading a known workload to a custom chip can also save a lot on operations costs, particularly power. Facebook is interested in workload operations per watt rather than raw floating point operations per watt. A GPU might have better raw specs but if the whole GPU package has worse workload ops per watt, a custom chip is likely better.

At Facebook's scale the spherical cow raw performance stats don't matter nearly as much as real world workloads per ops dollar. They can also repurpose their GPUs to other workloads and let their custom chips handle the boring baseline stuff.

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#52

Post by prpl »

rfoo wrote: post #37
thorncorona wrote: post #26

Earlier quoted context omitted.

WDYM by HPC mindset?

"The only meaningful benchmark in the world is LAPACK and only larger than ever monolithic problem instances matter, I don't know what you're talking about, 'embarrassingly parallel'? What a silly word! Serving web requests concurrently ? Good for you, congratulations, but can you do parallel programming?" Sorry if this make anyone feels bad. It certainly made myself uncomfortable typing it out though.

Roughly this. Part of it is performance fetish. Part of it is one architecture for every purpose. I can’t tell you how many times I’ve seen people run embarrassingly parallel jobs coordinated by MPI on a Cray - because somebody spent all that money on that machine. Don’t forget about Bell prize outages.

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#54

Post by sangnoir »

janalsncm wrote: post #50
tony_cannistra wrote: post #12

Earlier quoted context omitted.

Don't mean to single you out at all, but I find this comment to be a great example of how the "ML Hype" is perceived by a certain segment folks in our industry. The development of this chip shows that it doesn't (and shouldn't!) matter to the ML teams at Meta how 'fast ML is evolving.' Indeed what it demonstrates is that a huge, global, trillion-dollar business has operationalized an existing ML technology to the ext…

In their defense, it’s because the article is (understandably) sparse on details about what makes the requirements of their ranking models different from image classification or LLMs. Unless you work in industry it’s unlikely you will have heard of DeepFM or ESMM or whatever Meta is using. And building out specialized hardware does lock you in to a certain extent. Want to use more than 128GB of memory? Too bad, your…

> Want to use more than 128GB of memory? Too bad, your $10B chip doesn’t support that.

Which is probably why Meta is also buying the biggest Nvidia datacenter cards by the shipload. There is no need to run inference for a small model - say for a text-ad recommendation system - on an H100 with attendant electricity and cooling costs.

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#55

Post by samspenc »

janalsncm wrote: post #46
leetharris wrote: post #3

Earlier quoted context omitted.

They are different architectures optimized for different things. From the Meta post: "This chip’s architecture is fundamentally focused on providing the right balance of compute, memory bandwidth, and memory capacity for serving ranking and recommendation models." Optimizing for ranking/recommendation models is very different from general purpose training/inference.

Translation: you don’t need to serve 96 layer transformers for ranking and recommendation. You’re probably using a neural net with around 10-20 million parameters. But it needs to be fast and highly parallelizable, and perhaps perform well in lower precisions like f16. And it would be great to have a very large vector LUT on the same chip.

Is there a better way to compare performance across these high-end chips? The only comparable numbers I was able to find were the TFLOPS.

Meta seems to be reported these numbers for this v2 chip:

    708 TFLOPS/s (INT8) (sparsity)
    354 TFLOPS/s (INT8)
And I see Nvidia reporting these numbers for its latest Blackwell chips https://www.anandtech.com/show/21310/nvidia-blackwell-archit...

    4500 T(FL)OPS INT8/FP8 Tensor
Am I understanding correctly that Nvidia's upcoming Blackwell chips are 5-10x faster than this one Meta just announced?

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#56

Post by bevekspldnw »

Pretty fascinating they mention applications for ad serving but not Metaverse.

I feel like Zuck figured out he’s just running an ads network, the world is a long way anway from some VR fever dream, and to focus on milking each DAU for as many clicks as possible.

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#58

Post by ipsum2 »

samspenc wrote: post #55
janalsncm wrote: post #46

Earlier quoted context omitted.

Translation: you don’t need to serve 96 layer transformers for ranking and recommendation. You’re probably using a neural net with around 10-20 million parameters. But it needs to be fast and highly parallelizable, and perhaps perform well in lower precisions like f16. And it would be great to have a very large vector LUT on the same chip.

Is there a better way to compare performance across these high-end chips? The only comparable numbers I was able to find were the TFLOPS. Meta seems to be reported these numbers for this v2 chip: 708 TFLOPS/s (INT8) (sparsity) 354 TFLOPS/s (INT8) And I see Nvidia reporting these numbers for its latest Blackwell chips https://www.anandtech.com/show/21310/nvidia-blackwell-archit... 4500 T(FL)OPS INT8/FP8 Tensor Am I un…

To a rough approximation, yes. The blackwell chip is also ~10x larger in surface area than MTIA, so the costs are proportional.

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#59

Post by ec109685 »

bevekspldnw wrote: post #56

Pretty fascinating they mention applications for ad serving but not Metaverse. I feel like Zuck figured out he’s just running an ads network, the world is a long way anway from some VR fever dream, and to focus on milking each DAU for as many clicks as possible.

It’s not a gpu, and these chips aren’t able to generate images fast enough at inference time to be usable in VR context.

Re: Meta MTIA v2 – Meta Training and Inference Accelerator

#60

Post by photonbeam »

bevekspldnw wrote: post #56

Pretty fascinating they mention applications for ad serving but not Metaverse. I feel like Zuck figured out he’s just running an ads network, the world is a long way anway from some VR fever dream, and to focus on milking each DAU for as many clicks as possible.

Hes always known what pays the bills
Post reply on HN