Is parallel programming hard, and, if so, what can you do about it?
paulmck.livejournal.com
Is parallel programming hard, and, if so, what can you do about it?
1–10 of 88 posts
Re: Is parallel programming hard, and, if so, what can you do about it?
#2Re: Is parallel programming hard, and, if so, what can you do about it?
#3Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its performance implications, new design patterns (e.g. async/await, promises and futures). But hey, programming has never been thought of as easy, right? Yet, speaking of having it hard, it may be enlightening to compare software development with having to design a piece of (digital) hardware, where not only much has always been happening "in parallel" but one also must keep taking into account physical characteristics of various parts of the circuit.
Re: Is parallel programming hard, and, if so, what can you do about it?
#4Parallel programming in Python is hard. On Windows, it's hell.
In Matlab it's a breeze. Just put "par" in front of the thing. Done.
If you wanna shell out money to enable that functionality, that is.
Re: Is parallel programming hard, and, if so, what can you do about it?
#5A thread from 2015: https://news.ycombinator.com/item?id=9315152 2014: https://news.ycombinator.com/item?id=7381877 2011: https://news.ycombinator.com/item?id=2784515
Thanks!
FWIW, I've used the original post title ("Parallel Programming: December 2019 Update") to focus on the updates (and partially driven by the fear of comments made right after reading just the book title, "Is Parallel Programming Hard, And, If So, What Can You Do About It?", and focusing solely on answering the question while completely ignoring the content... :-]).
Re: Is parallel programming hard, and, if so, what can you do about it?
#6Someone wiser than me observed, “the thing about parallel programming is that it always ends in tears.”
Re: Is parallel programming hard, and, if so, what can you do about it?
#7Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…
> Parallel programming is not hard per se, it's just that there's so much more to learn
Parallel programming isn't the hardest part (it is hard, but good teachers help with good abstractions over problems), it is the parallel debugging that is - or at least the second pass over the same codebase.
Often being able to write decent parallel code is easier than debugging a system which is behaving inconsistently without any ability to set up the timing bugs the right way around.
The difference between "I fixed a bug" and "I massively lowered the probability of it" is hard to tell.
I spent 7 months chasing an issue that happened once on average of a billion requests (or rather once every 2-3 days across a web farm). The fix was easy, but debugging it drove me insane slowly over a whole summer.
And after that, there's the loss of efficiency from parallelization which needs enough parallel tracks to actually be faster - more threads is often slower at low thread counts.
As an anecdote, there were enough servers at Yahoo which had SMT turned off because the extra locking introduced by 2 threads on a core ate up the extra CPU those threads freed up.
Re: Is parallel programming hard, and, if so, what can you do about it?
#8It's hard or easy depending on your model (of parallelism) and what you want to do with it.
From a certain POV computers are always doing things in parallel (8-, 16-, 32-, 64-bits at a time, and so on, eh?)
Has anyone mentioned Pi Calculus yet?
Re: Is parallel programming hard, and, if so, what can you do about it?
#9It's hard or easy depending on your model (of parallelism) and what you want to do with it. From a certain POV computers are always doing things in parallel (8-, 16-, 32-, 64-bits at a time, and so on, eh?) Has anyone mentioned Pi Calculus yet? https://en.wikipedia.org/wiki/Pi-calculus
Anecdote: for me the actor model has been the most understandable and useful concurrency primitive I've used. Pi-calculus, which was inspired by the actor model, is similarly elegant.
Re: Is parallel programming hard, and, if so, what can you do about it?
#10One version of the parallel vs concurrency distinction is parallelism is the easy part and concurrency is the hard part.
I would say that for concurrency, FRP (or various incremental programming technics) is essential. The problem is the possible control flow / traces space is just too large to characterize by hand. Another problem is control flow can be bidirectional. If you join together two streams and wish to be push driven, when you reach the merge you need to be able to pull from the other side. Virtually all of Futures/Tokio, hit JS thing du hour, etc etc fail at this. Even forks are a bit hard to do manual CPS as you only have one continuation. Blocking calls fail because the inflexibility of linear 1 input 1 output, but this is better known.