Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]

cs.rice.edu

Post reply on HN
Print view View original

11–14 of 14 posts

Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]

#11

Post by nickpsecurity »

solidasparagus wrote: post #9

Interesting work! This is really an engineering achievement and I wish there was usable code. Real-time checkpointing seems like obviously the future to me, but it's going to be an easy-to-use, high-performance implementation that make that reality. One of the things I would like to have seen in the paper is a better analysis of simply checkpointing more often. It's briefly touched on: > It is infeasible to arbitrari…

It's strange because supercomputing centers have long built compute and storage in parallel to address this problem. Older companies like SGI had the storage accessing the high-speed, low-latency interconnect. Others build clusters with different nodes for each.

Companies that can train models this big should hire people with HPC experience. They'd point out the need for storage clusters with high-speed interconnects. If they lack storage capabilities, I wonder why they're doing HPC like that. They clearly need the storage.

Example that BLOOM was trained on lists 100+GB of RAM per node and PB's of storage:

http://www.idris.fr/eng/jean-zay/cpu/jean-zay-cpu-hw-eng.htm...

Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]

#12

Post by gwern »

Heh. This problem reminds me of back in 2019 when I was working with Shawn Presser on finetuning GPT-2 using Google Colab - there was a problem where it would randomly error out every once in a while, but also it would take like 10 minutes to redownload the last saved checkpoint from our server IIRC and it would take minutes to save the current checkpoint, so the question was, how often should we save to minimize the time spent restoring+saving? I did a bit of algebra and I think we wound up with an answer like '40 minutes'!

DL infrastructure & training practices have gotten better since then...

Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]

#13

Post by solidasparagus »

nickpsecurity wrote: post #11
solidasparagus wrote: post #9

Interesting work! This is really an engineering achievement and I wish there was usable code. Real-time checkpointing seems like obviously the future to me, but it's going to be an easy-to-use, high-performance implementation that make that reality. One of the things I would like to have seen in the paper is a better analysis of simply checkpointing more often. It's briefly touched on: > It is infeasible to arbitrari…

It's strange because supercomputing centers have long built compute and storage in parallel to address this problem. Older companies like SGI had the storage accessing the high-speed, low-latency interconnect. Others build clusters with different nodes for each. Companies that can train models this big should hire people with HPC experience. They'd point out the need for storage clusters with high-speed interconnects…

And it's gotten easier. On AWS where the paper was done, you can very easily get managed lustre or use S3 which can achieve very high bandwidth.
Post reply on HN