Live data from Hacker News

Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs

engineering.fb.com

Post reply on HN

Print view View original

1–3 of 3 posts

Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs

#1

Post by TheGuyWhoCodes » Fri, Jul 16, 2021, 10:14 PM UTC

Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs
engineering.fb.com

Re: Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs

#2

Post by gwern » Sat, Jul 17, 2021, 2:03 AM UTC

So, ZeRO-Offload?

Re: Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs

#3

Post by sksq96 » Thu, Jul 22, 2021, 11:00 AM UTC

gwern wrote: post #2

So, ZeRO-Offload?

ZeRO-Offload and ZeRO-3 and GPipe.

Post reply on HN