11–20 of 116 posts
How fast are they compared to raw memory throughput? It's interesting that memory mapping is so expensive. I've often wondered the price that everyone pays for multiple address spades. Is isolation really worth it?
I've known this one for awhile, and it makes writing shell scripts that glue two (or more) things together with pipes to do extremely high performance operations both rewarding and hilarious. Certainly one of the most useful tools in the toolbox.
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
AFAIK a severe limitation of pipes is that they can buffer only 64 KB / 16 pages (on x86 Linux). Pretty sure it's generally slower than core-to-memory bandwidth.
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
This is why threads aren't nearly as important as many programmers seem to think. Chances are, whatever application you're building can be done in a cleaner way using pipes + processes or green/user-space threads depending on the workload in question. It can be less convenient , but message passing is usually preferable to deadlock hell.
TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…
This is why threads aren't nearly as important as many programmers seem to think. Chances are, whatever application you're building can be done in a cleaner way using pipes + processes or green/user-space threads depending on the workload in question. It can be less convenient , but message passing is usually preferable to deadlock hell.
Threads are an important structuring mechanism: You can assume that all your threads continue to run, or in the event of a crash, all your threads die.
Also, unidirectional pipes aren't exactly sufficient for inter-process / inter-thread synchronisation. They are ok for simple batch processing, but that's about it.