Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Parallelism Blues: when faster code is slower

pythonspeed.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: The Parallelism Blues: when faster code is slower

#11

Post by winkeltripel »

_bxg1 wrote: post #10

None of this is surprising, right? Unless your system has fewer threads than cores (which it probably doesn't even without your program) there will always be some context-switching overhead. It's worth keeping in mind I guess - especially the fact that numpy parallelizes transparently - but generally these results are to be expected. The title is also misleading; it suggests that the wall clock time might be longer f…

> the wall clock time might be longer for parallel code

That is exactly the case, if CPU is the bottleneck in your already-parallel application. It's a case where we really shouldn't be layering different parallel bits together in one codebase, but might be doing it naively.

Re: The Parallelism Blues: when faster code is slower

#12

Post by jfim »

twoodfin wrote: post #9
maweki wrote: post #3

"It would be extremely surprising, then, if running with N threads actually gave ×N performance." Basically impossible by Ahmdal's law.

Of course in the general case Amdahl's law is inescapable, but some tasks on modern systems can show > ×N speedup over single-threaded performance if, for example, a single thread can only exploit at maximum some fraction of the total memory bandwidth or some level of the cache hierarchy.

Do you have examples where running n threads instead of 1 results in a speedup greater than n?

The only thing I can think of would be that the additional threads would kick the CPU into using a higher frequency, but a single thread using 100% of the CPU should already do that.

Re: The Parallelism Blues: when faster code is slower

#14

Post by The_rationalist »

eigenspace wrote: post #6
The_rationalist wrote: post #5

Earlier quoted context omitted.

Do you know if Julia will add OpenMP support? It's clearly the way to go for offloading to hardware in a productive way.

I don't know about "clearly the way to go". I think Julia's parallelism models have proven themselves to be very robust, performant and composeable, moreso than OpenMP as far as I'm aware.

How can I annotate an existing loop to offload it on the GPU Inclusive OR on AVX IOR on cpu cores. Without this ability, in practice I use far less parallelism.

Re: The Parallelism Blues: when faster code is slower

#15

Post by CUViper »

jfim wrote: post #12
twoodfin wrote: post #9

Earlier quoted context omitted.

Of course in the general case Amdahl's law is inescapable, but some tasks on modern systems can show > ×N speedup over single-threaded performance if, for example, a single thread can only exploit at maximum some fraction of the total memory bandwidth or some level of the cache hierarchy.

Do you have examples where running n threads instead of 1 results in a speedup greater than n? The only thing I can think of would be that the additional threads would kick the CPU into using a higher frequency, but a single thread using 100% of the CPU should already do that.

There are some broad examples of that effect here: https://en.wikipedia.org/wiki/Speedup#Super-linear_speedup

Re: The Parallelism Blues: when faster code is slower

#16

Post by dagw »

The_rationalist wrote: post #5
longemen3000 wrote: post #2

In Julia, where the paralleization options are explicit (SIMD, AVX, threads or multiprocessing), it always depends on the load, for small operation (around 10000 elements) a single thread is faster only for the thread spawning time (around 1 microsecond). And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads... In a nutshell, parallelization is no…

Do you know if Julia will add OpenMP support? It's clearly the way to go for offloading to hardware in a productive way.

Julia is actually initially had OpenMP backed parallelism (ParallelAccelerator.jl), but they're moving away from OpenMP towards a novel and native task parallelism framework more inspired by things like Cilk[0].

[0] https://julialang.org/blog/2019/07/multithreading/

Re: The Parallelism Blues: when faster code is slower

#17

Post by dagw »

The_rationalist wrote: post #14
eigenspace wrote: post #6

Earlier quoted context omitted.

I don't know about "clearly the way to go". I think Julia's parallelism models have proven themselves to be very robust, performant and composeable, moreso than OpenMP as far as I'm aware.

How can I annotate an existing loop to offload it on the GPU Inclusive OR on AVX IOR on cpu cores. Without this ability, in practice I use far less parallelism.

This is currently no official solution in Julia that I'm aware of. However there are several people working on it and a few experimental solution are under active development

https://github.com/JuliaDiffEq/AutoOffload.jl

https://juliagpu.gitlab.io/GPUifyLoops.jl/

Re: The Parallelism Blues: when faster code is slower

#18

Post by ChrisRackauckas »

dagw wrote: post #17
The_rationalist wrote: post #14

Earlier quoted context omitted.

How can I annotate an existing loop to offload it on the GPU Inclusive OR on AVX IOR on cpu cores. Without this ability, in practice I use far less parallelism.

This is currently no official solution in Julia that I'm aware of. However there are several people working on it and a few experimental solution are under active development https://github.com/JuliaDiffEq/AutoOffload.jl https://juliagpu.gitlab.io/GPUifyLoops.jl/

AutoOffload is something different, where it's trying to do linear algebra in a way that auto-offloads to GPUs or heterogeneous. GPUifyLoops is correct for this answer, and its next incarnation is KernelAbstractions.jl. These auto-construct GPU kernels and such from loops.

Re: The Parallelism Blues: when faster code is slower

#19

Post by ChrisRackauckas »

longemen3000 wrote: post #2

In Julia, where the paralleization options are explicit (SIMD, AVX, threads or multiprocessing), it always depends on the load, for small operation (around 10000 elements) a single thread is faster only for the thread spawning time (around 1 microsecond). And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads... In a nutshell, parallelization is no…

> And there is the issue of the independent Blas threaded model, where the Blas threads sometimes interfere with Julia threads

Julia has composible multithreading, and using that model fixed composing FFTW threads with Julia's. This can be done to OpenBLAS as well, and IIRC there is a PR open for it.

Re: The Parallelism Blues: when faster code is slower

#20

Post by The_rationalist »

dagw wrote: post #16
The_rationalist wrote: post #5

Earlier quoted context omitted.

Do you know if Julia will add OpenMP support? It's clearly the way to go for offloading to hardware in a productive way.

Julia is actually initially had OpenMP backed parallelism (ParallelAccelerator.jl), but they're moving away from OpenMP towards a novel and native task parallelism framework more inspired by things like Cilk[0]. [0] https://julialang.org/blog/2019/07/multithreading/

Cilk was officially deprecated in favor of openMP. But if it's just about inspiration, OK.
Post reply on HN