Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AITemplate, a revolutionary new inference engine by Meta AI

ai.facebook.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: AITemplate, a revolutionary new inference engine by Meta AI

#31

Post by PresentHarmony »

ipiszy wrote: post #29
PresentHarmony wrote: post #27

Earlier quoted context omitted.

Do I get it right, it takes 0.55 second or 0.49 second to generate an image depending on the batch? Thank you so much for your post! I would be very grateful for the response!

Yes this is correct. batch 16 7.9s / 25 steps, per image 0.49s: it generates 16 images for each prompt within 7.9s, so it's 0.49s per image.

One more question, if you don't mind. 1 image is generated in 0.7 seconds (25 steps ) and the same single image with 50 steps will be generated in 1.3 seconds. So it's much cheaper to generate more images for the same promt. Am I right or am I missing something ? Thanks in advance for your answer.

P.S. Though it should be 1.4 seconds. 0.7*2=14.If you think twice the speps, twice the time.

Re: AITemplate, a revolutionary new inference engine by Meta AI

#32

Post by tveita »

haolu7 wrote: post #8

AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution. More benchmark numbers and repro at: https://github.com/facebookincub…

This lists "OOM" for PyTorch on a RTX 3080-10GB, but I believe people have optimized the PyTorch SD model to run on even 6GiB GPUs.

Would AITemplate be able to run with those constraints?

Re: AITemplate, a revolutionary new inference engine by Meta AI

#33

Post by papersnake »

ipiszy wrote: post #7

tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…

Have you tested this on big models involving multi-gpu communication, or any plans?

Re: AITemplate, a revolutionary new inference engine by Meta AI

#34

Post by fooblaster »

ipiszy wrote: post #7

tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…

How does the performance compare with tensor rt? I didn't see any benchmarks comparing against that. I expect it to be lower for now, but excited for see what the future brings.

Re: AITemplate, a revolutionary new inference engine by Meta AI

#35

Post by ipiszy »

tveita wrote: post #32
haolu7 wrote: post #8

AITemplate-PyTorch Stable Diffusion is the fastest stable diffusion inference solution by pushing image generation below one second on A100 (batch 1: 0.7s / 25 steps, 1.3s / 50 steps; batch 3: 1.6s / 25 steps, per image 0.55s; batch 16 7.9s / 25 steps, per image 0.49s) for the first time, 2.57X faster than Keras' XLA-based GPU compilation solution. More benchmark numbers and repro at: https://github.com/facebookincub…

This lists "OOM" for PyTorch on a RTX 3080-10GB, but I believe people have optimized the PyTorch SD model to run on even 6GiB GPUs. Would AITemplate be able to run with those constraints?

RTX 3080-10GB should work. You could check https://github.com/facebookincubator/AITemplate/tree/main/ex..., and https://www.reddit.com/r/StableDiffusion/comments/xv7m89/met....

Re: AITemplate, a revolutionary new inference engine by Meta AI

#36

Post by ipiszy »

papersnake wrote: post #33
ipiszy wrote: post #7

tl;dr: Meta is open sourcing AITemplate, an inference engine for both Nvidia and AMD GPUs. Code: https://github.com/facebookincubator/AITemplate . AITemplate delivers much better perf (1.9x ~ 12.8x) compared to PyTorch eager on SOTA models, including Bert, ResNet, VIT and StableDiffusion. AITemplate also delivers high perf numbers using AMD GPUs (MI-250). With AITemplate, MI-250 achieves 80% ~ 96% A100 perf on variou…

Have you tested this on big models involving multi-gpu communication, or any plans?

For now it's for single GPU inference only.
Post reply on HN