DualPipe: Bidirectional pipeline parallelism algorithm
1–10 of 23 posts
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#2Hmm weren’t there also supposed to be the SM re-allocation, doesn’t look like it was included; I may have been mis-remembering the explanation.
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#3I attached all 3 algorithms 1F1B (1 forward 1 backward), ZB1P (zero bubble pipeline parallelism) and DualPipe as a picture here: https://x.com/danielhanchen/status/1894937006352031832 for those interested :)
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#4I attached all 3 algorithms 1F1B (1 forward 1 backward), ZB1P (zero bubble pipeline parallelism) and DualPipe as a picture here: https://x.com/danielhanchen/status/1894937006352031832 for those interested :)
Maybe add Chimera as well?
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#5I attached all 3 algorithms 1F1B (1 forward 1 backward), ZB1P (zero bubble pipeline parallelism) and DualPipe as a picture here: https://x.com/danielhanchen/status/1894937006352031832 for those interested :)
Off topic, but this is the Rick and Morty episode where Rick creates a perfectly level space.
The symmetry is uuugh.
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#6Sorry for us utter simpletons can someone explain what it do?
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#7I attached all 3 algorithms 1F1B (1 forward 1 backward), ZB1P (zero bubble pipeline parallelism) and DualPipe as a picture here: https://x.com/danielhanchen/status/1894937006352031832 for those interested :)
Maybe add Chimera as well? https://arxiv.org/pdf/2107.06925
Oh more nice pictures :)
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#8I attached all 3 algorithms 1F1B (1 forward 1 backward), ZB1P (zero bubble pipeline parallelism) and DualPipe as a picture here: https://x.com/danielhanchen/status/1894937006352031832 for those interested :)
Off topic, but this is the Rick and Morty episode where Rick creates a perfectly level space. The symmetry is uuugh .
You'll have to refresh my memory :) Is there like a Youtube clip for it?
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#9> DualPipe was created and developed by Jiashi Li and Chengqi Deng and Wenfeng Liang.
A CEO who codes.
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#10Earlier quoted context omitted.
Off topic, but this is the Rick and Morty episode where Rick creates a perfectly level space. The symmetry is uuugh .
You'll have to refresh my memory :) Is there like a Youtube clip for it?