Tldr?
Unfortunately your comment was too long. I didn’t read it. Try being more succinct next time.
AITemplate, a revolutionary new inference engine by Meta AI
21–30 of 36 posts
Re: AITemplate, a revolutionary new inference engine by Meta AI
#22tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…
How did AITemplate performance to state of art inference engine like tvm or onnx runtime ? Did AITemplate optimize/quantify network? Edit: link for TVM https://tvm.apache.org/
I'd love to hear about this too: especially after running the model through an onnx optimizer, like this one [0].
Re: AITemplate, a revolutionary new inference engine by Meta AI
#23AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution. More benchmark numbers and repro at: https://github.com/facebookincub…
Can you please eloborate, how many milliseconds does it take to generate 1 image with these wonderful improvements? I will be very grateful for your answer! Thank you very much!
Re: AITemplate, a revolutionary new inference engine by Meta AI
#24AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution. More benchmark numbers and repro at: https://github.com/facebookincub…
Or if you count in another way. In one second, how many pictures it will be able to generate, with these parameters. It could be 1.05, 1.1, or say 1.5 or even 2 pictures. Thank you very much for your post! I will be very grateful for the answer!
Re: AITemplate, a revolutionary new inference engine by Meta AI
#25tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…
Do you know of any good explanations of the techniques you used for those who only touch PyTorch Eager + occasionally torchscript?
You could check "AITemplate optimizations" section in the blog (https://ai.facebook.com/blog/gpu-inference-engine-nvidia-amd...), and https://github.com/facebookincubator/AITemplate#more-about-a.... The basic idea is to do aggressive kernel fusions.
Re: AITemplate, a revolutionary new inference engine by Meta AI
#26tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…
How do you verify the correctness of your fusion operation ?
We have a bunch of unittests and E2E tests to compare numeric numbers between AITemplate and PyTorch eager.
Re: AITemplate, a revolutionary new inference engine by Meta AI
#27AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution. More benchmark numbers and repro at: https://github.com/facebookincub…
Do I get it right, it takes 0.55 second or 0.49 second to generate an image depending on the batch?
Thank you so much for your post! I would be very grateful for the response!
Re: AITemplate, a revolutionary new inference engine by Meta AI
#28tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…
How did AITemplate performance to state of art inference engine like tvm or onnx runtime ? Did AITemplate optimize/quantify network? Edit: link for TVM https://tvm.apache.org/
AITemplate only supports fp16 data types with fp16 or fp32 accumulation right now. We are working on supporting more data types and quantization.
We don't have an official comparison between AITemplate and tvm / onnx for now, but we do have perf numbers like https://github.com/facebookincubator/AITemplate/tree/main/ex..., https://github.com/facebookincubator/AITemplate/tree/main/ex.... Feel free to run these examples on other frameworks and compare perf.
Re: AITemplate, a revolutionary new inference engine by Meta AI
#29AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution. More benchmark numbers and repro at: https://github.com/facebookincub…
Do I get it right, it takes 0.55 second or 0.49 second to generate an image depending on the batch? Thank you so much for your post! I would be very grateful for the response!
Yes this is correct.
batch 16 7.9s / 25 steps, per image 0.49s: it generates 16 images for each prompt within 7.9s, so it's 0.49s per image.
Re: AITemplate, a revolutionary new inference engine by Meta AI
#30Earlier quoted context omitted.
Do I get it right, it takes 0.55 second or 0.49 second to generate an image depending on the batch? Thank you so much for your post! I would be very grateful for the response!
Yes this is correct. batch 16 7.9s / 25 steps, per image 0.49s: it generates 16 images for each prompt within 7.9s, so it's 0.49s per image.
Thank you indeed, my friend!