AITemplate, a revolutionary new inference engine by Meta AI
ai.facebook.com
AITemplate, a revolutionary new inference engine by Meta AI
1–10 of 36 posts
Re: AITemplate, a revolutionary new inference engine by Meta AI
#2Re: AITemplate, a revolutionary new inference engine by Meta AI
#3https://github.com/facebookincubator/AITemplate
Very impressive results!
Re: AITemplate, a revolutionary new inference engine by Meta AI
#4It supports both Nvidia and AMD, and both got pretty good speedup. This is a great achievement!
Re: AITemplate, a revolutionary new inference engine by Meta AI
#5Tldr?
Re: AITemplate, a revolutionary new inference engine by Meta AI
#6For all the hate that Facebook gets their only redeeming quality is these open source projects they have been releasing all along.
Maybe this is to attract better engineers but all in all this has been a net postive for software development. So credit where it is due.
Re: AITemplate, a revolutionary new inference engine by Meta AI
#7tl;dr:
Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate.
AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion.
AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on various ResNet / Bert / VIT models.
AITemplate uses sophisticated fusion techniques to optimize perf, including vertical, horizontal, and memory fusions.
btw, I'm one of the authors of AITemplate, happy to answer any questions.
Re: AITemplate, a revolutionary new inference engine by Meta AI
#8AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution.
More benchmark numbers and repro at: https://github.com/facebookincubator/AITemplate/tree/main/ex...
Re: AITemplate, a revolutionary new inference engine by Meta AI
#9Sadly it doesn't have Apple GPU backend
Re: AITemplate, a revolutionary new inference engine by Meta AI
#10tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…
Thanks, that is very helpful. Do you have to train the model differently for use with AITemplate? Could it be helpful for Leela Chess Zero (LC0)? I think LC0 has a generic Pytorch backend, that is several times slower than its NVidia specific CUDA backend. I'm not very clueful about this stuff though.