Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

mirrors.edge.kernel.org

Post reply on HN
Print view View original

121–130 of 199 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#121

Post by vore »

crabbone wrote: post #111
ascar wrote: post #101

Earlier quoted context omitted.

Well, funnily enough this does read in contrast to the definitions used in Wikipedia, which are the ones I am also familiar with (I also do teach a class called "Parallel Programming" to graduates). I do think the differentation make sense from a perspective of problem classes, as also evident from the comments here. Running independent problems in parallel to better utilize hardware ressources is very different from…

The definition found in Wikipedia, like many contentious subjects in programming, was written by people with strong political agenda and very little respect to the matter being described. This applies to all sorts of ambiguous terms used very generously in the witchcraft of "applied computer science". Other examples include "object-oriented programming", "statically- or dynamically-typed language", "interpreted langu…

Uhh, I think it is pretty commonly accepted that a statically-typed language has typechecking facilities before runtime and a dynamically-typed language doesn't. Maybe there is a spectrum and things somewhere in the middle gradual typing but the general idea is quite clear.

Re: Is parallel programming hard, and, if so, what can you do about it?

#122

Post by crabbone »

bob1029 wrote: post #59
remontoire wrote: post #27

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas. What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough. Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude.…

This is a wrong view of the problem. Often times your application has to be distributed for reasons other than speed: there are only so many PCIe devices you can connect to a single CPU, there are only so many CPU sockets you can put on a single PCB and so on.

In large systems, parallel / concurrent applications are the baseline. If you have to replicate your data as its being generated into geographically separate location there's no way you can do it in a single thread...

Re: Is parallel programming hard, and, if so, what can you do about it?

#123

Post by cpgxiii »

phkahler wrote: post #114
crabbone wrote: post #106

Earlier quoted context omitted.

I don't know who invented this nonsense distinction. First time I was introduced to this idea of "concurrent programming" being a separate thing when Go was released. So, I associate this nonsense with Go, but it could have happened earlier, I simply never heard about it before then. Anyways. The way I see it used today, it's applied to language runtimes incapable or severely crippled when it comes to parallel / conc…

>> don't know who invented this nonsense distinction ... It not nonsense. In C or C++ lots of code can be made parallel using OpenMP and inserting some #pragma statements above for loops. This does not work for things like running a UI in one thread and some other work in another thread, perhaps displaying results as they are found. These are quite different types of parallelism.

OpenMP versus std::thread/pthread/etc isn't this distinction. Both are thread-based parallelism; that OpenMP implementations do so cleverly via a threadpool approach is secondary. Any loop you can write using OpenMP you could dispatch manually via an explicit thread-based approach, and long-running parallel task you'd normally implement via std::async/std::future/std::thread could be dispatched via OpenMP. There are good conventions about what kinds of operations are better expressed via one or the other mechanism, but they are effectively equivalent in capability.

Re: Is parallel programming hard, and, if so, what can you do about it?

#124

Post by ascar »

Nathanba wrote: post #117
phkahler wrote: post #114

Earlier quoted context omitted.

>> don't know who invented this nonsense distinction ... It not nonsense. In C or C++ lots of code can be made parallel using OpenMP and inserting some #pragma statements above for loops. This does not work for things like running a UI in one thread and some other work in another thread, perhaps displaying results as they are found. These are quite different types of parallelism.

I think it's nonsense to invent new terminology for this tiny minor difference in how you use threads and whether you wait on results or you don't wait.

but it's just not a tiny minor difference. The problems you end up dealing with are entirely different.

The terminology used might be terrible since they're originally synonyms. Differentiating between the two is very useful though.

Re: Is parallel programming hard, and, if so, what can you do about it?

#125

Post by AmpsterMan »

Nathanba wrote: post #117
phkahler wrote: post #114

Earlier quoted context omitted.

>> don't know who invented this nonsense distinction ... It not nonsense. In C or C++ lots of code can be made parallel using OpenMP and inserting some #pragma statements above for loops. This does not work for things like running a UI in one thread and some other work in another thread, perhaps displaying results as they are found. These are quite different types of parallelism.

I think it's nonsense to invent new terminology for this tiny minor difference in how you use threads and whether you wait on results or you don't wait.

I'm sorry but you are betraying a myopic view here. Concurrency as a computer science concept has existed at least since the existence of co-routines which have been around for 60+ years. Parallelism as a concept has also existed for about as long. I find it fascinating that you take issue with "concurrency" when that one is the older concept in the engineering zeitgeist. If anything, parallelism is the odd one because it only became sounding average engineers started thinking about when consumer CPUs started becoming multi threaded.

Re: Is parallel programming hard, and, if so, what can you do about it?

#126

Post by taeric »

ascar wrote: post #79
taeric wrote: post #75

Earlier quoted context omitted.

I am curious on that. 500 million events per second sounds high. Even for games. That many calculations? Sure. I take "events" to mean user generated, though. And that sounds high. Same for searches. Difficulty there is size of search space, not searches coming in. Right?

Google search isn't a good example. AAA games are a great example when you think about graphics. However, most of that is trivially parallelizable, thus "all you need to do" is assign vertices/pixels to different threads (in quotation marks as that's of course not trivial by itself, but a different kind of engineering problem). However, once you get into simulations you have billions (or multiple orders of magnitude…

Thinking of graphics, though, i would assume most of that is in the GPU side. Simulations do make sense, but I see games like Factorio still focused on single thread first. And then look for natural parallel segments.

That is all to say that millions of events still feels like a lot. I am not shocked to know it can and does happen.

Re: Is parallel programming hard, and, if so, what can you do about it?

#127

Post by Nathanba »

ascar wrote: post #124
Nathanba wrote: post #117

Earlier quoted context omitted.

I think it's nonsense to invent new terminology for this tiny minor difference in how you use threads and whether you wait on results or you don't wait.

but it's just not a tiny minor difference. The problems you end up dealing with are entirely different. The terminology used might be terrible since they're originally synonyms. Differentiating between the two is very useful though.

how are the problems different? In all cases you want the maximum performance. The only decision here is how you schedule your tasks. Ideally entirely in parallel and you only schedule them to wait if you can't find a way to make them 100% parallel

Re: Is parallel programming hard, and, if so, what can you do about it?

#128

Post by aidenn0 »

ascar wrote: post #84

I see a lot of confusion between parallel programming [1] and concurrent programming [2] in the comments here. The former and what this book is about deals with the problem of parallelizaing a single sequential program. There usually is strong interaction or dependencies between elements and progress needs synchronization. E.g. timestep iterations in real-time simulations that need synchronization with data communica…

Both I (and apparently the author of TFA) disagree with your definition of parallel programming. TFA gives an example of "embarrassingly parallel" programs as one way to make parallel programming simple.

The distinction I learned was: any time you have multiple logical threads of execution you have concurrency, any time you have multiple computations happening simultaneously, you have parallelism.

Multithreaded programming on a single core computer is concurrent, but not parallel. Vector processing is parallel, but not concurrent.

Re: Is parallel programming hard, and, if so, what can you do about it?

#129

Post by nmarinov »

remontoire wrote: post #27

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

Fun video[0]. The optimization bit starts at 0:35. [0]: https://www.youtube.com/watch?v=VgSQ1GOC86s

Re: Is parallel programming hard, and, if so, what can you do about it?

#130

Post by ascar »

elil17 wrote: post #95
OscarCunningham wrote: post #91

Earlier quoted context omitted.

So this might be a naive question, but is it literally that you're simulating a fluid in parallel by having each thread simulate a portion of the space? And then the message passing between threads is the data about the fluid on the boundary?

Computational fluid dynamics is actually a problem where parallelization doesn't get you much. It is primarily limited by memory bandwidth.

Memory bandwidth can be increased by parallelization though. E.g. MPI (Message Passing Interface) is one of the major libraries in parallel programming and supercomputing and deals with parallelization across multiple machines.
Post reply on HN