DualPipe: Bidirectional pipeline parallelism algorithm • Hacker News Board

Re: DualPipe: Bidirectional pipeline parallelism algorithm

#21

Post by larodi » Thu, Feb 27, 2025, 2:26 PM UTC

anonzzzies wrote: post #17

xnhbx wrote: post #9

> DualPipe was created and developed by Jiashi Li and Chengqi Deng and Wenfeng Liang. A CEO who codes.

When my company was still working closely with CN factories a few years ago (before the bans / clients no longer wanting to work with companies working with china etc), the CEO's of the factories we worked with all were electronic engineers at that company or another before; they all could jump in, debug schematics, sold and write firmware themselves. And they did. These were places with massive campuses with towerin…

It sounds so more reasonable to have a director who is actually technical, doesn't it? I'm absolutely amazed how this (to the east) contrasts to understanding (to the west) that directors rather need to know finance, strategic planning, and marketing, than the actual nuance of the work.

Re: DualPipe: Bidirectional pipeline parallelism algorithm

#22

Post by tway223 » Thu, Feb 27, 2025, 2:52 PM UTC

larodi wrote: post #21

anonzzzies wrote: post #17

Earlier quoted context omitted.

When my company was still working closely with CN factories a few years ago (before the bans / clients no longer wanting to work with companies working with china etc), the CEO's of the factories we worked with all were electronic engineers at that company or another before; they all could jump in, debug schematics, sold and write firmware themselves. And they did. These were places with massive campuses with towerin…

It sounds so more reasonable to have a director who is actually technical, doesn't it? I'm absolutely amazed how this (to the east) contrasts to understanding (to the west) that directors rather need to know finance, strategic planning, and marketing, than the actual nuance of the work.

To be blunt this is exactly what is wrong with the “leadership” mindset in the west, as decisions are often made without understanding the “nuances” yet they are confident it would work.

Re: DualPipe: Bidirectional pipeline parallelism algorithm

#23

Post by qrios » Thu, Feb 27, 2025, 7:56 PM UTC

puppycodes wrote: post #6

Sorry for us utter simpletons can someone explain what it do?

In very simple words: it is one way to reduce the white squares in the picture from @danielhanchen[1].

In more complex words: imagine a processor which is able to process every instruction in 10 clock cycles. But also the processor is able to get new input for this instruction on every clock cycle and starts to process this new input in a pipeline. After the first input you have to wait ten clock cycles. But if you feed the input line every time you will get the output also permanently.

In the case of GPUs, it is now not only a topic of a single pipeline, but multiple in parallel. Depends on your data and algorithm it can be thousands in parallel.

[1] https://x.com/danielhanchen/status/1894937006352031832