Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
1–10 of 14 posts
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#2Using an original name will probably make it easier for people to find your paper.
Maybe ask ChatGPT for ideas.
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#3For anyone else who was confused to see a paper use the same name as a commercial product, it looks like Google Gemini was announced in May, whereas this was submitted to SOSP that had an April submission deadline.
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#4Very interesting. Well done authors!
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#5Using an original name will probably make it easier for people to find your paper. Maybe ask ChatGPT for ideas.
We should let Google know.
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#6For anyone else who was confused to see a paper use the same name as a commercial product, it looks like Google Gemini was announced in May, whereas this was submitted to SOSP that had an April submission deadline.
I think this points more to how slow the paper submission process is compared to the product creation velocity. No wonder arxiv has been such a hit for the ML community.
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#7For anyone else who was confused to see a paper use the same name as a commercial product, it looks like Google Gemini was announced in May, whereas this was submitted to SOSP that had an April submission deadline.
I think this points more to how slow the paper submission process is compared to the product creation velocity. No wonder arxiv has been such a hit for the ML community.
along with also a flood of subpar publications
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#8For anyone else who was confused to see a paper use the same name as a commercial product, it looks like Google Gemini was announced in May, whereas this was submitted to SOSP that had an April submission deadline.
It's not a good name to give to anything. Unless you're a corporate giant, name creativity is really important to making your work findable and re-findable.
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#9Interesting work! This is really an engineering achievement and I wish there was usable code. Real-time checkpointing seems like obviously the future to me, but it's going to be an easy-to-use, high-performance implementation that make that reality.
One of the things I would like to have seen in the paper is a better analysis of simply checkpointing more often. It's briefly touched on:
> It is infeasible to arbitrarily increase the checkpoint frequency because checkpoint frequency is bottlenecked by the bandwidth of the remote persistent storage [28]. For example, it takes 42 minutes to checkpoint the model states of MT-NLG [68] to the remote persistent storage when the bandwidth is 20Gbps.
and
> Both baselines, Strawman and HighFreq, have the same checkpoint time and it stays almost the same as the number of machines increases from 4 to 16 because the aggregated bandwidth of the remote persistent storage is fixed
But that smells a bit off to me. That's a 530B model (unrealistically large given current trends IMO) where each model replica has 280 A100s and then there is data parallelism on top. Where exactly are you storing your (sharded?) checkpoint where the read/write bandwidth isn't also scaling horizontally beyond 20Gbps?
Re: Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints [pdf]
#10For anyone else who was confused to see a paper use the same name as a commercial product, it looks like Google Gemini was announced in May, whereas this was submitted to SOSP that had an April submission deadline.
It's not a good name to give to anything. Unless you're a corporate giant, name creativity is really important to making your work findable and re-findable.
> name creativity is really important to making your work findable and re-findable.
This is underrated information. I’ve seen so many products and even companies fail because the name led to millions of unrelated search results. Even if they are a giant it can still lead to bad outcomes.