Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

paulmck.livejournal.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#41

Post by imtringued »

Understand the fundamental problems/principles.

Almost all problems and solutions stem from one limitation: only one writer at any given time. Immutable means one writer. Locking means one writer. Splitting a list and processing each half means one writer. The actor model means one writer. All are valid solutions to your problems.

A piece of personal advice: Avoid acquiring more than one lock at the same time if possible. The solutions are usually incredibly messy. The dining philosopher problem is a pretty good example. Every single problem involving multiple "per object" locks requires a hand crafted solution to the problem. Changing the order in which forks are picked up does not generalize well to other problems.

If you have global locks you may get away with always acquiring them in the same order but if you make a single mistake you're opening yourself up to deadlocks.

Re: Is parallel programming hard, and, if so, what can you do about it?

#42

Post by keymone »

rubyn00bie wrote: post #24

Oh parallelism, the one thing I used to worry about more, and probably should (as far as using GPUs goes)... but also one I don't really sweat so much. Most of the reason, for me, was finding Elixir/Erlang AND their virtual machine BEAM. Until Elixir and BEAM for me, parallel programming had been difficult because the competing mental models in other languages pretty much make for a textbook fantasy. I'm sure some fo…

you should learn about clojure. not only it provides nice concurrency semantics (not actors though, more granular stuff), it's also functional (in a pragmatic way, like erlang) and immutable to the core.

tutorial: https://www.braveclojure.com/introduction/

on gpu: https://neanderthal.uncomplicate.org/articles/tutorial_openc...

Re: Is parallel programming hard, and, if so, what can you do about it?

#43

Post by dragandj »

keymone wrote: post #42
rubyn00bie wrote: post #24

Oh parallelism, the one thing I used to worry about more, and probably should (as far as using GPUs goes)... but also one I don't really sweat so much. Most of the reason, for me, was finding Elixir/Erlang AND their virtual machine BEAM. Until Elixir and BEAM for me, parallel programming had been difficult because the competing mental models in other languages pretty much make for a textbook fantasy. I'm sure some fo…

you should learn about clojure. not only it provides nice concurrency semantics (not actors though, more granular stuff), it's also functional (in a pragmatic way, like erlang) and immutable to the core. tutorial: https://www.braveclojure.com/introduction/ on gpu: https://neanderthal.uncomplicate.org/articles/tutorial_openc...

While we're on Clojure and the GPU, let me shamelessly promote my upcoming books (I'm the author of that old tutorial and the state of GPU dev in Clojure progressed a lot since then):

Deep Learning for Programmers: An Interactive Tutorial with CUDA, OpenCL, DNNL, Java, and Clojure

https://aiprobook.com/deep-learning-for-programmers/

Numerical Linear Algebra for Programmers: An Interactive Tutorial with GPU, CUDA, OpenCL, MKL, Java, and Clojure

https://aiprobook.com/numerical-linear-algebra-for-programme...

Both books progress really well, and you can subscribe now to read the drafts and get the complete books when ready. There's no middle man and 100% of proceeds goes into funding the development of open source Clojure HPC/GPU/ML/DL libraries.

Re: Is parallel programming hard, and, if so, what can you do about it?

#44

Post by cryptica »

imtringued wrote: post #41

Understand the fundamental problems/principles. Almost all problems and solutions stem from one limitation: only one writer at any given time. Immutable means one writer. Locking means one writer. Splitting a list and processing each half means one writer. The actor model means one writer. All are valid solutions to your problems. A piece of personal advice: Avoid acquiring more than one lock at the same time if poss…

These are good tips but I would also add that if you can avoid shared memory altogether, then you should avoid it.

If you can shard and distribute the data neatly and evenly so that each process handles a completely separate subset, then that is the most reliable long term solution.

With shared memory, it's difficult to control who has access to what parts and when - and that can make it difficult to detect and analyse bottlenecks.

Re: Is parallel programming hard, and, if so, what can you do about it?

#45

Post by gpderetta »

joe_the_user wrote: post #23
shaklee3 wrote: post #21

I feel like every comment here simply read the title, and not even skimmed the book. This is a book about extreme low-level optimizations in concurrency. It's a poor title for this kind of book since it seems like it's a beginner book, but it's quite the opposite. It's arguing it's hard to implement the structures and methods used by concurrency APIs, not that it's hard to use them.

It doesn't help that the link goes to a page that links to another page that links to the book and the first page doesn't make it clear that that it's talking about a book.

is there an HTML version?

Re: Is parallel programming hard, and, if so, what can you do about it?

#46

Post by diegoperini »

nurettin wrote: post #36
joe_the_user wrote: post #22

Earlier quoted context omitted.

> Parallel programming is not hard per se I'm not sure how you can say that. Programming a fairly hard activity but the general consensus is adding parallelism to any given programming task, you make that task much, much harder. I don't think you find any version of parallel programming that isn't hard. The situation of parallelism, your commands happening in an unpredictable order, not synchronously, inherently, by…

>> I don't think you find any version of parallel programming that isn't hard. Just serialize, pass data onto multiple processes, synchronize and wait on all of them to return their data back to main process. This is one kind of parallel programming that isn't hard. But I cheatead and created expensive copies and processes and let the OS handle all the nightmare.

That's not a cheat, just a solution with some trade-offs.

Re: Is parallel programming hard, and, if so, what can you do about it?

#47

Post by DaiPlusPlus »

gopalv wrote: post #7
Koshkin wrote: post #3

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…

> Parallel programming is not hard per se, it's just that there's so much more to learn Parallel programming isn't the hardest part (it is hard, but good teachers help with good abstractions over problems), it is the parallel debugging that is - or at least the second pass over the same codebase. Often being able to write decent parallel code is easier than debugging a system which is behaving inconsistently without…

> Parallel programming isn't the hardest part [...] it is the parallel debugging that is - or at least the second pass over the same codebase.

We need better IDE parallel thread debugging tools. I don’t like how VS won’t let you open a separate Autos/Locals window for each thread, or easily step each thread - setting a breakpoint in a function currently executing in multiple threads is a usability nightmare.

Re: Is parallel programming hard, and, if so, what can you do about it?

#48

Post by verttii »

gleenn wrote: post #20
Syzygies wrote: post #15

Haskell. I'm at almost fifty years of programming (APL was an early highlight; punched cards weren't) in many languages. Haskell isn't the be all / end all of languages, but when I want to add ten lines to a program and have it use every core, I don't know a better choice than Haskell. Nothing else comes close.

I came here to say this except for Clojure. While not as pure as Haskell, it has amazing parallel programming primitives like atoms, pmap, software transactional memory, and more that are extremely easy to us relative to trying to accomplish the same thing in Java etc in my very humble opinion.

Isn't Software Transactional Memory more for concurrency than parallel computing though?

Re: Is parallel programming hard, and, if so, what can you do about it?

#49

Post by verttii »

k_bx wrote: post #31
Polylactic_acid wrote: post #18

Earlier quoted context omitted.

I used Haskell for a few months. It seemed like a really well designed language. I didn't look too much about the multi threading stuff (Does it just work automatically for all haskell code?) The main issue I found was the language and libraries are really hard to pick up as a beginner. A library for ruby or python would start off with a few examples showing how the most common use case works and then full api docs f…

This is a totally valid point. There are now initiatives like https://tech.fpcomplete.com/haskell that gather tutorials for essential stuff, but I think we're far from being beginner-friendly. I don't think there's any other option except for improving libraries little by little, adding well-written tutorials.

Haskell absolutely lacks the soft documentation needed to attract new people. Just about every library could use a "mini tutorial" in their README, at the very least, to allow even novices to quickly bootstrap them and start getting productive without having to understand all the underlying concepts.

Re: Is parallel programming hard, and, if so, what can you do about it?

#50

Post by gpderetta »

brazzy wrote: post #37
joe_the_user wrote: post #22

Earlier quoted context omitted.

> Parallel programming is not hard per se I'm not sure how you can say that. Programming a fairly hard activity but the general consensus is adding parallelism to any given programming task, you make that task much, much harder. I don't think you find any version of parallel programming that isn't hard. The situation of parallelism, your commands happening in an unpredictable order, not synchronously, inherently, by…

> I don't think you find any version of parallel programming that isn't hard. Using pure functional programming on an embarassingly parallel workload comes pretty close.

trivially parallel workload is trivial in pretty much any language though.
Post reply on HN