Does this remind anyone else of the Pied Piper compression algorithm?
DualPipe: Bidirectional pipeline parallelism algorithm
11–20 of 23 posts
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#12I attached all 3 algorithms 1F1B (1 forward 1 backward), ZB1P (zero bubble pipeline parallelism) and DualPipe as a picture here: https://x.com/danielhanchen/status/1894937006352031832 for those interested :)
Maybe add Chimera as well? https://arxiv.org/pdf/2107.06925
it looks as if Chimera has marginally less bubbles than DualPipe?
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#13[dead]
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#14Sorry for us utter simpletons can someone explain what it do?
It makes it so that having more GPUs makes inference run faster. Worst case has been you can only use memory from them and gain no speed at all
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#15I hope all the open sources Deepseek is doing encourages American labs to do more of the same. Surely they'll realize their momentum is more of a moat than their tech at any one point in time.
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#16Re: DualPipe: Bidirectional pipeline parallelism algorithm
#17> DualPipe was created and developed by Jiashi Li and Chengqi Deng and Wenfeng Liang. A CEO who codes.
When my company was still working closely with CN factories a few years ago (before the bans / clients no longer wanting to work with companies working with china etc), the CEO's of the factories we worked with all were electronic engineers at that company or another before; they all could jump in, debug schematics, sold and write firmware themselves. And they did. These were places with massive campuses with towering buildings with robots and a few (relative to the massive space) employees doing maintenance etc + prototyping.
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#18Does this remind anyone else of the Pied Piper compression algorithm?
Middle out or something?
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#19> DualPipe was created and developed by Jiashi Li and Chengqi Deng and Wenfeng Liang. A CEO who codes.
"developed" and "codes" have different meanings.
Re: DualPipe: Bidirectional pipeline parallelism algorithm
#20> DualPipe was created and developed by Jiashi Li and Chengqi Deng and Wenfeng Liang. A CEO who codes.
"developed" and "codes" have different meanings.
Yes but in this context, they are very close to each other in meaning.
Besides Liang does indeed code a significant amount and has contributed to almost all of their published papers.