Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

mirrors.edge.kernel.org

Post reply on HN
Print view View original

191–199 of 199 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#191

Post by elil17 »

uguuo_o wrote: post #167
elil17 wrote: post #95

Earlier quoted context omitted.

Computational fluid dynamics is actually a problem where parallelization doesn't get you much. It is primarily limited by memory bandwidth.

I am not sure where you get this information. Parallelization is everything in this space, hence why we have highly interconnected supercomputers to model the most difficult engineering problems. Typical runs use 30k+ cores for a single problem for weeks on end [0]. There are some special cases, such as Boltzmann/dvm [1] where invididual partitions of cells have millions of degrees of freedom, where memory bandwidth…

Those large number of cores come with a diminishing return (see this benchmark, for example: https://nusit.nus.edu.sg/services/hpc-newsletter/cfd-simulat...)

Re: Is parallel programming hard, and, if so, what can you do about it?

#193

Post by aidenn0 »

Salgat wrote: post #175
aidenn0 wrote: post #128

Earlier quoted context omitted.

Both I (and apparently the author of TFA) disagree with your definition of parallel programming. TFA gives an example of "embarrassingly parallel" programs as one way to make parallel programming simple. The distinction I learned was: any time you have multiple logical threads of execution you have concurrency, any time you have multiple computations happening simultaneously, you have parallelism. Multithreaded progr…

Vector processing is not parallelism, but rather non-scalar. Specifically, it's a single operation that is able to do work on multiple data items, rather than parallel processors doing work at the same time.

It is data parallel. In the same way, I would label super-scalar CPUs machines that automatically perform parallel processing on a linear stream of instructions (taking advantage of so-called "instruction level parallelism."

Re: Is parallel programming hard, and, if so, what can you do about it?

#194

Post by chlorion »

ascar wrote: post #189
chlorion wrote: post #176

Earlier quoted context omitted.

Yes, concurrency on a single core CPU is simply not possible. That's why multitasking OS's didn't exist until multicore CPUs became a thing. If only these "Gophers" knew anything about computing history!

So before 2001 your OS didn't have a scheduler and was incapable of handling more than one process at the same time? The user interface was simply hanging when you told your CPU to do something?

That was the point I was attempting to get across. Single core computers multitask fine, which proves that concurrency is not the same thing as parallelism.

Even in modern times, we have something like the first generation raspberry pi zero, it has a single ARM core, and it multitasks fine.

Re: Is parallel programming hard, and, if so, what can you do about it?

#195

Post by gpderetta »

tialaramex wrote: post #139
gpderetta wrote: post #113

Earlier quoted context omitted.

yet physical hardware has a finite amount of memory, so at some point you either stop your program of you have to modify a cell you already touched.

Mutation isn't confusing. On its own. Multiple references isn't confusing. On its own. The observation, which I don't think I saw being made twenty years ago (I could be wrong) is that you shouldn't mix these two things. No multiple references with mutation. In theory you can safely do so in serial programs if you were careful enough, in practice you won't be careful enough and we should write fewer serial programs t…

My point was simply that at some point the buck stops and someone has to write the code to mutate a memory cell.

Regarding your point, shared xor mutable is nice, but databases are an example of a shared mutable (and even concurrent!) yet safe resource, so other models are possible.

Re: Is parallel programming hard, and, if so, what can you do about it?

#196

Post by uguuo_o »

elil17 wrote: post #191
uguuo_o wrote: post #167

Earlier quoted context omitted.

I am not sure where you get this information. Parallelization is everything in this space, hence why we have highly interconnected supercomputers to model the most difficult engineering problems. Typical runs use 30k+ cores for a single problem for weeks on end [0]. There are some special cases, such as Boltzmann/dvm [1] where invididual partitions of cells have millions of degrees of freedom, where memory bandwidth…

Those large number of cores come with a diminishing return (see this benchmark, for example: https://nusit.nus.edu.sg/services/hpc-newsletter/cfd-simulat... )

A poorly-optimized code that does not scale is not evidence that all simulation tools behave the same. Most of the tools in use by NASA, DOD, DOE, research institutions and commercial codes scale very well. Weak and strong scaling both. In fact, scalability is one of the primary requirements for simulation tools that are used in mission critical environments. I've been the technical lead for many of these types of projects, and I have experience with most of the largest commercial, research, and open source simulation codes. The vast majority of parallel tools scale linearly well beyond the 10s of thousands of cores, and I do agree that at some point adding cores can cause bottleneck, but that point is usually far past the few hunder cores from your link for a significant portion of engineering applications.

It is common knowledge in the field to optimize a simulation for the largest number of cores it can efficiently use, so simulation cases are not just blindly thrown more cores without a justification given by the scalability. Your initial claim that parallelization doesn't get you much is flawed, parallelization is the only thing that enables scalable engineering analysis.

Re: Is parallel programming hard, and, if so, what can you do about it?

#197

Post by kllrnohj »

OkayPhysicist wrote: post #159
kllrnohj wrote: post #43

Earlier quoted context omitted.

You only ever need this if you're trying to have hundreds of thousands to millions of threads. It's a very niche problem to have

It's a niche problem to have because our current programming paradigm treats concurrency as a second-class citizen. The invariants of most languages do not include those required for mass concurrency. Functional programming better aligns with the requirements, which is how you arrive at Erlang and Elixir. Every map() function can be trivially replaced with the concurrent cmap(), because the side effects that would ma…

Except in that scenario you absolutely don't want green threads but real threads. M:N threading (especially if N=1) doesn't help you get parallelism, and parallelism is what you need for modern CPUs.

So that again keeps green threads (and thus mass concurrency without parallelism) in the niche category.

Re: Is parallel programming hard, and, if so, what can you do about it?

#198

Post by mrkeen »

gpderetta wrote: post #195
tialaramex wrote: post #139

Earlier quoted context omitted.

Mutation isn't confusing. On its own. Multiple references isn't confusing. On its own. The observation, which I don't think I saw being made twenty years ago (I could be wrong) is that you shouldn't mix these two things. No multiple references with mutation. In theory you can safely do so in serial programs if you were careful enough, in practice you won't be careful enough and we should write fewer serial programs t…

My point was simply that at some point the buck stops and someone has to write the code to mutate a memory cell. Regarding your point, shared xor mutable is nice, but databases are an example of a shared mutable (and even concurrent!) yet safe resource, so other models are possible.

> databases are an example

Yes, it is possible because programmers have accepted the bargain that they must write purely-functional SQL.

Then the db is able to decide how best to optimise/apply/retry/copy-on-write/abort as necessary.

If SQL allowed you to directly mutate things, it would break everything. You couldn't 'LIMIT 10' if your loop might want to increment some data in row 40082.

Re: Is parallel programming hard, and, if so, what can you do about it?

#199

Post by tm-guimaraes »

bsdetector wrote: post #55
tm-guimaraes wrote: post #13

Earlier quoted context omitted.

That’s only safety. It still does not guarantee that performance would improve.

A built-in parallel foreach can't guarantee better performance because the hardware isn't there. The parallel part is scheduled on normal threads by the kernel and this is too unpredictable. But if each OS-scheduled CPU was really a cluster of for instance 1 fast + 64 slow cores then it could have special instructions to split a low-level loop on these slow cores and to facilitate blocking and rejoining back into a s…

That's pretty much the usual GPU+CPU workflow... Only you would have less badwith and latency issues.

Still, deciding where to run such things are not trivial, the time it takes to decide what would be faster could actually be more than the time it takes to just run it.

Post reply on HN