The Parallelism Blues: when faster code is slower
pythonspeed.com
The Parallelism Blues: when faster code is slower
1–10 of 25 posts
Re: The Parallelism Blues: when faster code is slower
#2In Julia, where the paralleization options are explicit (SIMD, AVX, threads or multiprocessing), it always depends on the load, for small operation (around 10000 elements) a single thread is faster only for the thread spawning time (around 1 microsecond). And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads...
In a nutshell, parallelization is not a magical bullet, but is a good bullet to have at your disposal anyway
Re: The Parallelism Blues: when faster code is slower
#3"It would be extremely surprising, then, if running with N threads actually gave ×N performance."
Basically impossible by Ahmdal's law.
Re: The Parallelism Blues: when faster code is slower
#4This is perfectly normal behavior when Intel Hyperthreading is involved.
I'm on my phone, so rather than trying to type out an explanation, I'm going to link to Wikipedia: https://en.wikipedia.org/wiki/Hyper-threading
Re: The Parallelism Blues: when faster code is slower
#5In Julia, where the paralleization options are explicit (SIMD, AVX, threads or multiprocessing), it always depends on the load, for small operation (around 10000 elements) a single thread is faster only for the thread spawning time (around 1 microsecond). And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads... In a nutshell, parallelization is no…
Do you know if Julia will add OpenMP support? It's clearly the way to go for offloading to hardware in a productive way.
Re: The Parallelism Blues: when faster code is slower
#6In Julia, where the paralleization options are explicit (SIMD, AVX, threads or multiprocessing), it always depends on the load, for small operation (around 10000 elements) a single thread is faster only for the thread spawning time (around 1 microsecond). And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads... In a nutshell, parallelization is no…
Do you know if Julia will add OpenMP support? It's clearly the way to go for offloading to hardware in a productive way.
I don't know about "clearly the way to go". I think Julia's parallelism models have proven themselves to be very robust, performant and composeable, moreso than OpenMP as far as I'm aware.
Re: The Parallelism Blues: when faster code is slower
#7This is perfectly normal behavior when Intel Hyperthreading is involved. I'm on my phone, so rather than trying to type out an explanation, I'm going to link to Wikipedia: https://en.wikipedia.org/wiki/Hyper-threading
The benchmarks were run on a computer with hyperthreading disabled.
Re: The Parallelism Blues: when faster code is slower
#8"It would be extremely surprising, then, if running with N threads actually gave ×N performance." Basically impossible by Ahmdal's law.
well before that kicks in, if your code requires any coordination at all (not Monte Carlo) then those overheads can scale with the number of processes. so rather than hitting an asymptote as you'd get with just Ahmdal, you actually start to go down in absolute terms as the number of processes increases.
Re: The Parallelism Blues: when faster code is slower
#9"It would be extremely surprising, then, if running with N threads actually gave ×N performance." Basically impossible by Ahmdal's law.
Of course in the general case Amdahl's law is inescapable, but some tasks on modern systems can show > ×N speedup over single-threaded performance if, for example, a single thread can only exploit at maximum some fraction of the total memory bandwidth or some level of the cache hierarchy.
Re: The Parallelism Blues: when faster code is slower
#10None of this is surprising, right? Unless your system has fewer threads than cores (which it probably doesn't even without your program) there will always be some context-switching overhead. It's worth keeping in mind I guess - especially the fact that numpy parallelizes transparently - but generally these results are to be expected.
The title is also misleading; it suggests that the wall clock time might be longer for parallel code in certain cases. While not impossible, that isn't what the article covers.