Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

mirrors.edge.kernel.org

Post reply on HN
Print view View original

171–180 of 199 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#171

Post by Salgat »

ascar wrote: post #84

I see a lot of confusion between parallel programming [1] and concurrent programming [2] in the comments here. The former and what this book is about deals with the problem of parallelizaing a single sequential program. There usually is strong interaction or dependencies between elements and progress needs synchronization. E.g. timestep iterations in real-time simulations that need synchronization with data communica…

Parallel programming is simply a type of concurrent programming. Concurrent means that two tasks can both progress in a given duration. On a single core computer every thread runs concurrently. Parallel expands on concurrency to mean that two concurrent tasks can also run at the exact same time. On a multi-core computer threads can run in parallel. In many cases concurrent programming and parallel programming have little to no difference, and you program with the assumption that every task can run in parallel (for example, whenever you use async/await or the threadpool).

Re: Is parallel programming hard, and, if so, what can you do about it?

#172

Post by saltcured »

crabbone wrote: post #106
ascar wrote: post #84

I see a lot of confusion between parallel programming [1] and concurrent programming [2] in the comments here. The former and what this book is about deals with the problem of parallelizaing a single sequential program. There usually is strong interaction or dependencies between elements and progress needs synchronization. E.g. timestep iterations in real-time simulations that need synchronization with data communica…

I don't know who invented this nonsense distinction. First time I was introduced to this idea of "concurrent programming" being a separate thing when Go was released. So, I associate this nonsense with Go, but it could have happened earlier, I simply never heard about it before then. Anyways. The way I see it used today, it's applied to language runtimes incapable or severely crippled when it comes to parallel / conc…

I'm not sure I can identify when it started, but these were already the concepts commonly in use when I did my CS undergraduate work in the early 90s. I.e. it was in textbooks and course titles as established jargon.

Concurrency was the kind of thing worried about in OS design or Unix programming styles whether on a time-sharing system or some small scale multi-processing system. Coordination of heterogeneous sequential programs on some shared resources.

Parallelism was the topic of high-performance computing with combined use of many hardware resources to accelerate a single algorithm.

Of course these are simplifying abstractions, and real systems can get into the murky gray area that is both concurrent and parallel.

Re: Is parallel programming hard, and, if so, what can you do about it?

#173

Post by vore »

Jtsummers wrote: post #163
vore wrote: post #158

Earlier quoted context omitted.

Single core concurrency doesn't have to deal with hardware memory synchronization.

> Single core concurrency doesn't have to deal with hardware memory synchronization. Yes it does. If you have two threads which (for whatever reason) are sharing memory and they can be run in an arbitrary order, you have to deal with memory synchronization. If it's a single core machine, you still need to synchronize your memory access. Mutexes and semaphores predate systems with multiple processors.

Yes, but those are not hardware memory synchronization mechanisms. You do not have to deal with memory fencing to coordinate reads/writes at the CPU level, because there is only one core that can read memory at a time.

Re: Is parallel programming hard, and, if so, what can you do about it?

#174

Post by vore »

CyberDildonics wrote: post #162
vore wrote: post #158

Earlier quoted context omitted.

Single core concurrency doesn't have to deal with hardware memory synchronization.

What point are you trying to make now? You just said: Parallelism is a physical thing, concurrency is a logical thing. So by your own definition, wouldn't multi-core concurrency be parallelism?

Yes? Parallelism is inherently concurrent but concurrency is not inherently parallel. But single core concurrency still exists as a form of non-parallel concurrency.

The point I'm trying to make is down to brass tacks, single core concurrency is a lot simpler than multi core concurrency, because you don't need any hardware cooperation.

Re: Is parallel programming hard, and, if so, what can you do about it?

#175

Post by Salgat »

aidenn0 wrote: post #128
ascar wrote: post #84

I see a lot of confusion between parallel programming [1] and concurrent programming [2] in the comments here. The former and what this book is about deals with the problem of parallelizaing a single sequential program. There usually is strong interaction or dependencies between elements and progress needs synchronization. E.g. timestep iterations in real-time simulations that need synchronization with data communica…

Both I (and apparently the author of TFA) disagree with your definition of parallel programming. TFA gives an example of "embarrassingly parallel" programs as one way to make parallel programming simple. The distinction I learned was: any time you have multiple logical threads of execution you have concurrency, any time you have multiple computations happening simultaneously, you have parallelism. Multithreaded progr…

Vector processing is not parallelism, but rather non-scalar. Specifically, it's a single operation that is able to do work on multiple data items, rather than parallel processors doing work at the same time.

Re: Is parallel programming hard, and, if so, what can you do about it?

#176

Post by chlorion »

crabbone wrote: post #141
vore wrote: post #110

Earlier quoted context omitted.

How is this nonsense or anything to do with "language runtimes with disabilities"? An OS running on a single core processor cannot be parallel but it may be concurrent: it can never physically do two things at the same time, but it might be able to logically interleave different tasks. Parallelism is a physical thing, concurrency is a logical thing.

You have just repeated the nonsense I was talking about. The claim you repeat is meaningless. A program is either parallel / concurrent or not. The situation you describe (when there's a single processor core) isn't parallel or concurrent. In some sense, it emulates concurrent / parallel execution because it imitates the unpredictable ordering of code execution, which sure has its uses... but the whole point of deali…

Yes, concurrency on a single core CPU is simply not possible. That's why multitasking OS's didn't exist until multicore CPUs became a thing.

If only these "Gophers" knew anything about computing history!

Re: Is parallel programming hard, and, if so, what can you do about it?

#177

Post by CyberDildonics »

vore wrote: post #174
CyberDildonics wrote: post #162

Earlier quoted context omitted.

What point are you trying to make now? You just said: Parallelism is a physical thing, concurrency is a logical thing. So by your own definition, wouldn't multi-core concurrency be parallelism?

Yes? Parallelism is inherently concurrent but concurrency is not inherently parallel. But single core concurrency still exists as a form of non-parallel concurrency. The point I'm trying to make is down to brass tacks, single core concurrency is a lot simpler than multi core concurrency, because you don't need any hardware cooperation.

Before you were saying they are two different things, now you're saying they are the same when you have multiple cores and not the same when you have a single core. If they're the same when you have multiple cores, isn't there just single and multi-core concurrency by your own definitions? If so, why are you making a distinction of parallelism and concurrency?

Re: Is parallel programming hard, and, if so, what can you do about it?

#178

Post by sn9 »

tambourine_man wrote: post #38
klabb3 wrote: post #5

I’m way-above-average interested in concurrent programming but this 600+ page brick will probably remain on my reading list until I am stranded on a deserted island. Did anyone here read the whole thing? Can one make a reasonable summary or is this more of a lexicon of different techniques?

Stranded on a deserted island is bit radical, but I recommend going to a place with no internet connection for a week or so. I though I had a long reading problem. Turns out, I have an internet problem.

The circuit in my apartment that my wifi is on keeps flipping so my connection has been frustratingly intermittent.

I've gotten more reading and cleaning done in the last week than I have in maybe years.

Re: Is parallel programming hard, and, if so, what can you do about it?

#179

Post by [deleted] »

uguuo_o wrote: post #167
elil17 wrote: post #95

Earlier quoted context omitted.

Computational fluid dynamics is actually a problem where parallelization doesn't get you much. It is primarily limited by memory bandwidth.

I am not sure where you get this information. Parallelization is everything in this space, hence why we have highly interconnected supercomputers to model the most difficult engineering problems. Typical runs use 30k+ cores for a single problem for weeks on end [0]. There are some special cases, such as Boltzmann/dvm [1] where invididual partitions of cells have millions of degrees of freedom, where memory bandwidth…

[deleted]

Re: Is parallel programming hard, and, if so, what can you do about it?

#180

Post by spacechild1 »

CyberDildonics wrote: post #157
spacechild1 wrote: post #85

Earlier quoted context omitted.

Don't forget about audio!

I don't think audio applies here at all. Each channel can be mixed in parallel, 44,100 samples per second is not much per channel and mixing isn't difficult. Also most can be cached and don't need to be mixed or filtered in real time because they haven't been changed from the last play.

Sorry, I should have been more specific. I meant DAWs and computer music environments, not simple audio players. Modern DAWs are heavily multi-threaded.
Post reply on HN