It's interesting that they are not separating training and inference.
Meta MTIA v2 – Meta Training and Inference Accelerator
41–50 of 63 posts
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#42Certainly an interesting looking chip. It looks like it's for recommendation workloads. Are those workloads very specific, or is there a possibility to run more general inference (image, language, etc) on this accelerator?
And, they mention a compiler in PyTorch, is that open sourced? I really liked the Google Coral chips -- they are perfect little chips for running image recognition and bounding box tasks. But since the compiler is closed source it's impossible to extend them for anything else beyond what Google had in mind for them when they came out in 2018, and they are completely tied to Tensorflow, with a very risky software support story going forward (it's a google product after all).
Is it the same story for this chip?
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#43[deleted]
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#44My mind still boggles that a BBS+ads company would think it needs to design its own chips.
"Depending on how you want to think about it, it was funny or inevitable or symbolic that the robotic takeover did not start at MIT, NASA, Microsoft or Ford. It started at a Burger-G restaurant ..." https://marshallbrain.com/manna1
https://www.ycombinator.com/companies/ofone/jobs/u2E2fCX-fou...
I saw this YC startup ad right after I finished reading this.
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#45Earlier quoted context omitted.
Yeah, it may fit their current workload perfectly, but it doesn't seem very future proof with the limited bandwidth. Given how fast ML is evolving these days I question if it makes sense to design and deploy a chip like this. I guess they do have a very large workload that will benefit immediately.
> Yeah, it may fit their current workload perfectly, but it doesn't seem very future proof It’s custom silicon designed for a specific, known workload. It’s not designed to be a general purpose part or to be future proofed for unknown future applications. When a new application comes along with new requirements, the teams will use their experience to create a new chip targeting that new application. That’s the great…
Right and they have a LOT of GPUs from Nvidia for handle the unknown. Custom silicon for custom workloads seems like a good strategy specially considering the capabilities that the team will develop along the way.
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#46Intel Gaudi 3 has more interconnect bandwidth than this has memory bandwidth. By a lot. I guess they can't be fairly compared without knowing the TCO for each. I know in the past Google's TPU per-chip specs lagged Nvidia but the much lower TCO made them a slam dunk for Google's inference workloads. But this seems pretty far behind the state of the art. No FP8 either.
They are different architectures optimized for different things. From the Meta post: "This chip’s architecture is fundamentally focused on providing the right balance of compute, memory bandwidth, and memory capacity for serving ranking and recommendation models." Optimizing for ranking/recommendation models is very different from general purpose training/inference.
Translation: you don’t need to serve 96 layer transformers for ranking and recommendation. You’re probably using a neural net with around 10-20 million parameters. But it needs to be fast and highly parallelizable, and perhaps perform well in lower precisions like f16. And it would be great to have a very large vector LUT on the same chip.
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#47It's interesting that they are not separating training and inference.
This is specifically designed for inference for recommendations models. It’s not for LLM training or inference.
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#48Come on, Zuck, undermine Google Cloud and take NVIDIA down a few pegs by offering this for purchase in good quantities.
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#49Intel Gaudi 3 has more interconnect bandwidth than this has memory bandwidth. By a lot. I guess they can't be fairly compared without knowing the TCO for each. I know in the past Google's TPU per-chip specs lagged Nvidia but the much lower TCO made them a slam dunk for Google's inference workloads. But this seems pretty far behind the state of the art. No FP8 either.
Also its at 90 watts vs 900 watts for gaudi 3, the flops/mem bw per watt is much more comparable.
With high end chips like that it's often possible to get dramatically better efficiency by running it at less than peak power consumption, like 90% performance at 50% power or something like that. It's hard to compare the numbers in a fair way.
Re: Meta MTIA v2 – Meta Training and Inference Accelerator
#50Earlier quoted context omitted.
Yeah, it may fit their current workload perfectly, but it doesn't seem very future proof with the limited bandwidth. Given how fast ML is evolving these days I question if it makes sense to design and deploy a chip like this. I guess they do have a very large workload that will benefit immediately.
Don't mean to single you out at all, but I find this comment to be a great example of how the "ML Hype" is perceived by a certain segment folks in our industry. The development of this chip shows that it doesn't (and shouldn't!) matter to the ML teams at Meta how 'fast ML is evolving.' Indeed what it demonstrates is that a huge, global, trillion-dollar business has operationalized an existing ML technology to the ext…
In their defense, it’s because the article is (understandably) sparse on details about what makes the requirements of their ranking models different from image classification or LLMs. Unless you work in industry it’s unlikely you will have heard of DeepFM or ESMM or whatever Meta is using.
And building out specialized hardware does lock you in to a certain extent. Want to use more than 128GB of memory? Too bad, your $10B chip doesn’t support that.