[deleted]
MTIA v1: Meta’s first-generation AI inference accelerator
21–30 of 50 posts
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#22It's curious why nobody is selling these systems yet
For the same reason why it took a long time for crypto mining accelerators to actually ship. It is more profitable to keep them for yourself.
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#23This looks like a customized ASIC specializing solely in recommendation systems possibly focused on ads ranking
>We found that GPUs were not always optimal for running Meta’s specific recommendation workloads at the levels of efficiency required at our scale. Our solution to this challenge was to design a family of recommendation-specific Meta Training and Inference Accelerator (MTIA) ASICs.
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#24Comparing MTIA v1 vs Google Cloud TPU v4: MTIA v1's specs: The accelerator is fabricated in TSMC 7nm process and runs at 800 MHz, providing 102.4 TOPS at INT8 precision and 51.2 TFLOPS at FP16 precision. It has a thermal design power (TDP) of 25 W. Up to 128 GB of ram LPDDR5. Googles Cloud TPU v4: 275 teraflops (bf16 or int8), 90/170/192 W. 32 GiB of HBM2 RAM, 1200 GBps. From here: https://cloud.google.com/tpu/docs/s…
FWIW, you're comparing a training-specialized chip to an inference-specialized chip. It'd be more apples to apples to compare to TPU v4 lite, but I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: https://arxiv.org/abs/2304.01433
How does a training specialized chip function? Forward mode is simple, just a dot product machine. But how do you accelerate backprop on hardware? Does it have the vector Jacobian transformation lookup logic and table baked into hardware?
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#25They designed it in 2020 does that mean it is likely to have been in use for a while or is the design lag a few years?
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#26They designed it in 2020 does that mean it is likely to have been in use for a while or is the design lag a few years?
It is ambiguous on that front. If you designed it in 2020, getting through test runs at TSMC and then to a final production run would take a while. So when they had it deployed at scale at FB is unclear.
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#27Has there been any rumors or statements from Facebook on them eventually stepping into selling cloud compute? I'd be surprised if they are investing in building hardware accelerators just for their own services.
Given that these chips seem to be power optimised and Facebook's recently released sensory model, I wouldn't be surprised to see them in their next iteration of VR devices.
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#28Just as incredible is the corresponding announcement of their RSC which is purportedly one of the world's most powerful clusters
Amazing times! Private companies now have compute resources previously only showing up in government labs, and in many cases using novel components like MTIA
This feels like the start of a golden age and in a few years we will have incredible results and breakthroughs
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#29Earlier quoted context omitted.
FWIW, you're comparing a training-specialized chip to an inference-specialized chip. It'd be more apples to apples to compare to TPU v4 lite, but I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: https://arxiv.org/abs/2304.01433
How does a training specialized chip function? Forward mode is simple, just a dot product machine. But how do you accelerate backprop on hardware? Does it have the vector Jacobian transformation lookup logic and table baked into hardware?
Look at the massive diff in the TDP & RAM to start with. Meta's is 1/3rd the TDP and has very different RAM.
Re: MTIA v1: Meta’s first-generation AI inference accelerator
#30Earlier quoted context omitted.
FWIW, you're comparing a training-specialized chip to an inference-specialized chip. It'd be more apples to apples to compare to TPU v4 lite, but I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: https://arxiv.org/abs/2304.01433
How does a training specialized chip function? Forward mode is simple, just a dot product machine. But how do you accelerate backprop on hardware? Does it have the vector Jacobian transformation lookup logic and table baked into hardware?
Mostly you need to be able to stash intermediate products computed in the forward phase so that you can access them in the backward phase. This requires more memory, more memory bandwidth, more transpose, and also, training usually operates at slightly higher precision (bf16 instead of int8 as one example).