Is parallel programming hard, and, if so, what can you do about it? • Hacker News Board

Re: Is parallel programming hard, and, if so, what can you do about it?

#51

Post by caspper69 » Mon, Jan 13, 2020, 1:01 PM UTC

clarry wrote: post #33

shaklee3 wrote: post #19

Earlier quoted context omitted.

Parallel programming is not hard if you follow the "just use a mutex and you avoid race conditions" mantra. However, this book dives deep into what mutexes actually do, what the hardware below is doing, and alternatives to a mutex for a speed increase. This can be anything from lockless algorithms to RCU, which the author of this book wrote it in Linux kernel. There's a comment above about Haskell and how you don't h…

> Parallel programming is not hard if you follow the "just use a mutex and you avoid race conditions" mantra. At one extreme of the spectrum, you can do everything under a big lock and be effectively single-threaded. At the other extreme, you can use lots of mutexes and fine grained locking and end up with a mess of potential deadlocks and races and "here be dragons" because the original programmer did not anticipate…

Well, parallel programming requires discipline, and a plan. Deadlocks do not occur when locks are always acquired in the same order. The fact that deadlocks occur at all (and I know they are notoriously hard to debug-- I've done it, lol), is due to the fact that one or both of the locking strategy and/or adherence to said locking strategy was flawed.

Re: Is parallel programming hard, and, if so, what can you do about it?

#52

Post by platz » Mon, Jan 13, 2020, 1:27 PM UTC

shaklee3 wrote: post #21

I feel like every comment here simply read the title, and not even skimmed the book. This is a book about extreme low-level optimizations in concurrency. It's a poor title for this kind of book since it seems like it's a beginner book, but it's quite the opposite. It's arguing it's hard to implement the structures and methods used by concurrency APIs, not that it's hard to use them.

It's hard to find problems that benefit from parallelism, because most work has sequential elements. Due to amdhal's law, it's harder to realize significant parallel gains unless the problem is of a special or constrained type.

Re: Is parallel programming hard, and, if so, what can you do about it?

#53

Post by brazzy » Mon, Jan 13, 2020, 1:31 PM UTC

gpderetta wrote: post #50

brazzy wrote: post #37

Earlier quoted context omitted.

> I don't think you find any version of parallel programming that isn't hard. Using pure functional programming on an embarassingly parallel workload comes pretty close.

trivially parallel workload is trivial in pretty much any language though.

Still easy to mess up the coordination when you have mutable shared state.

Re: Is parallel programming hard, and, if so, what can you do about it?

#54

Post by namibj » Mon, Jan 13, 2020, 1:55 PM UTC

agumonkey wrote: post #34

gopalv wrote: post #7

Earlier quoted context omitted.

> Parallel programming is not hard per se, it's just that there's so much more to learn Parallel programming isn't the hardest part (it is hard, but good teachers help with good abstractions over problems), it is the parallel debugging that is - or at least the second pass over the same codebase. Often being able to write decent parallel code is easier than debugging a system which is behaving inconsistently without…

This gives me two questions: - did you have a method/tool to debug (from book, or chatting with others) - are there still cpu designs dedicated to parallel workloads (I assume the usual desktop intel/amd, no matter how brilliant, may not be without hiccups in high parallelism) oh one last thing, do you use dedicated compilers for parallelization ? or is it something "mainstream" like openmp

I can greatly recommended mozilla's rr for hunting concurrency bugs, as it allows efficient replay and provides a scheduler that is designed to increase the probability of triggering concurrency bugs.

The issue with x86 is simple: the memory model pretty much requires loads of broadcasts on the SMP interconnect. IIRC Power9 is nice due to (1) a memory model that doesn't require this (much) broadcasting on the interconnect, and (2) remote atomics like "fetch current 64bit value pointed to by register1 and atomically add register2 after", which prevent the cacheline from bouncing between cores or even sockets (Power9 scales to about 5~10kW TDP @8~20W TDP/core and 4 threads/core (there are versions that combine core pairs to have 8 threads using the resources of (almost) 2 cores (overhead / diminishing returns))).

I can't really recommend OpenMP. If the architectural restrictions aren't a problem, you might want to check out Intel's TBB, which uses C++ templates instead of compiler pragmas. Unfortunately you'd ideally need multithreading-aware polyhedral optimization, and _that_ mostly stalled AFAIK.

You'd likely write C++ or now maybe Rust with low-level primitives / intrinsics to build libraries. I don't know of any actually good "automatic" frameworks for general shared-memory parallel programming (There's Chapel, but it seems fairly niche. Still going strong though, by the looks of it.). If you can efficiently express the problem with loop-dataflow, you might want to try timely-dataflow (Rust, does thread/process/node parallelism), as it nicely integrates with normal Rust code.

Re: Is parallel programming hard, and, if so, what can you do about it?

#55

Post by classified » Mon, Jan 13, 2020, 2:13 PM UTC

Direct jump to the PDF download page, avoiding the terrible hosting and Russian ads:

https://mirrors.edge.kernel.org/pub/linux/kernel/people/paul...

Re: Is parallel programming hard, and, if so, what can you do about it?

#56

Post by shaklee3 » Mon, Jan 13, 2020, 2:20 PM UTC

gpderetta wrote: post #45

joe_the_user wrote: post #23

Earlier quoted context omitted.

It doesn't help that the link goes to a page that links to another page that links to the book and the first page doesn't make it clear that that it's talking about a book.

is there an HTML version?

I believe there's only a PDF version, and the author encourages you to print it. It's a monumental effort to make that kind of book, so it's pretty amazing he released it free.

Re: Is parallel programming hard, and, if so, what can you do about it?

#57

Post by nurettin » Mon, Jan 13, 2020, 2:23 PM UTC

diegoperini wrote: post #46

nurettin wrote: post #36

Earlier quoted context omitted.

>> I don't think you find any version of parallel programming that isn't hard. Just serialize, pass data onto multiple processes, synchronize and wait on all of them to return their data back to main process. This is one kind of parallel programming that isn't hard. But I cheatead and created expensive copies and processes and let the OS handle all the nightmare.

That's not a cheat, just a solution with some trade-offs.

I was facetiously referring to not having to use any synchronization primitives, but apparently the meaning is lost in ink.

Re: Is parallel programming hard, and, if so, what can you do about it?

#58

Post by ColanR » Mon, Jan 13, 2020, 2:23 PM UTC

jonathanstrange wrote: post #35

joe_the_user wrote: post #22

Earlier quoted context omitted.

> Parallel programming is not hard per se I'm not sure how you can say that. Programming a fairly hard activity but the general consensus is adding parallelism to any given programming task, you make that task much, much harder. I don't think you find any version of parallel programming that isn't hard. The situation of parallelism, your commands happening in an unpredictable order, not synchronously, inherently, by…

I have to second that. Parallel programming is extremely hard to get right and people persistently underestimate it. For example, I've seen many times code that erroneously assumed that values can be read from global memory concurrently if only write access is guarded. Of course, it depends a lot on what your programming language has to offer. For example,in my experience Ada makes it easier to get parallelism correc…

> For example, I've seen many times code that erroneously assumed that values can be read from global memory concurrently if only write access is guarded.

As someone who has made that assumption, what's wrong with it?

Re: Is parallel programming hard, and, if so, what can you do about it?

#59

Post by jcranmer » Mon, Jan 13, 2020, 2:52 PM UTC

brazzy wrote: post #53

gpderetta wrote: post #50

Earlier quoted context omitted.

trivially parallel workload is trivial in pretty much any language though.

Still easy to mess up the coordination when you have mutable shared state.

Embarrassingly parallel workloads kinda don't have mutable shared state... that's why they're "embarrassing".

Re: Is parallel programming hard, and, if so, what can you do about it?

#60

Post by dreamcompiler » Mon, Jan 13, 2020, 2:59 PM UTC

imtringued wrote: post #41

Understand the fundamental problems/principles. Almost all problems and solutions stem from one limitation: only one writer at any given time. Immutable means one writer. Locking means one writer. Splitting a list and processing each half means one writer. The actor model means one writer. All are valid solutions to your problems. A piece of personal advice: Avoid acquiring more than one lock at the same time if poss…

> Avoid acquiring more than one lock at the same time if possible.

If you know you're going to need 2 locks and you acquire both of them at the same time atomically, deadlock is impossible. The trouble happens when you acquire one lock, wait a bit, then acquire a second one. Maybe that's what you meant and I just misread it.