Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

paulmck.livejournal.com

Post reply on HN
Print view View original

31–40 of 88 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#31

Post by k_bx »

Polylactic_acid wrote: post #18
Syzygies wrote: post #15

Haskell. I'm at almost fifty years of programming (APL was an early highlight; punched cards weren't) in many languages. Haskell isn't the be all / end all of languages, but when I want to add ten lines to a program and have it use every core, I don't know a better choice than Haskell. Nothing else comes close.

I used Haskell for a few months. It seemed like a really well designed language. I didn't look too much about the multi threading stuff (Does it just work automatically for all haskell code?) The main issue I found was the language and libraries are really hard to pick up as a beginner. A library for ruby or python would start off with a few examples showing how the most common use case works and then full api docs f…

This is a totally valid point. There are now initiatives like https://tech.fpcomplete.com/haskell that gather tutorials for essential stuff, but I think we're far from being beginner-friendly. I don't think there's any other option except for improving libraries little by little, adding well-written tutorials.

Re: Is parallel programming hard, and, if so, what can you do about it?

#32

Post by agumonkey »

Koshkin wrote: post #3

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…

relativistic factor: being taught that programming == sequential imperative turns your brain myopic, makes even harder to unlearn and think about concurrent/distributed systems

bonus point: to an extent sequential imperative is an abstraction layer over the concurrent IC implementing your computer

Re: Is parallel programming hard, and, if so, what can you do about it?

#33

Post by clarry »

shaklee3 wrote: post #19
Koshkin wrote: post #3

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…

Parallel programming is not hard if you follow the "just use a mutex and you avoid race conditions" mantra. However, this book dives deep into what mutexes actually do, what the hardware below is doing, and alternatives to a mutex for a speed increase. This can be anything from lockless algorithms to RCU, which the author of this book wrote it in Linux kernel. There's a comment above about Haskell and how you don't h…

> Parallel programming is not hard if you follow the "just use a mutex and you avoid race conditions" mantra.

At one extreme of the spectrum, you can do everything under a big lock and be effectively single-threaded. At the other extreme, you can use lots of mutexes and fine grained locking and end up with a mess of potential deadlocks and races and "here be dragons" because the original programmer did not anticipate that you sometimes need data not to change for a bit longer than a short and obvious critical section.

What I'm working on at work right now, oh goodness how many races and problems I've found around locking and multi-threading in general. It's a nightmare.

Re: Is parallel programming hard, and, if so, what can you do about it?

#34

Post by agumonkey »

gopalv wrote: post #7
Koshkin wrote: post #3

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…

> Parallel programming is not hard per se, it's just that there's so much more to learn Parallel programming isn't the hardest part (it is hard, but good teachers help with good abstractions over problems), it is the parallel debugging that is - or at least the second pass over the same codebase. Often being able to write decent parallel code is easier than debugging a system which is behaving inconsistently without…

This gives me two questions:

- did you have a method/tool to debug (from book, or chatting with others)

- are there still cpu designs dedicated to parallel workloads (I assume the usual desktop intel/amd, no matter how brilliant, may not be without hiccups in high parallelism)

oh one last thing, do you use dedicated compilers for parallelization ? or is it something "mainstream" like openmp

Re: Is parallel programming hard, and, if so, what can you do about it?

#35

Post by jonathanstrange »

joe_the_user wrote: post #22
Koshkin wrote: post #3

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…

> Parallel programming is not hard per se I'm not sure how you can say that. Programming a fairly hard activity but the general consensus is adding parallelism to any given programming task, you make that task much, much harder. I don't think you find any version of parallel programming that isn't hard. The situation of parallelism, your commands happening in an unpredictable order, not synchronously, inherently, by…

I have to second that. Parallel programming is extremely hard to get right and people persistently underestimate it. For example, I've seen many times code that erroneously assumed that values can be read from global memory concurrently if only write access is guarded.

Of course, it depends a lot on what your programming language has to offer. For example,in my experience Ada makes it easier to get parallelism correct than many other languages, primarily because Ada tasks have a precisely defined and specified semantics and because Ada's "Rendesvouz" makes certain kinds of errors impossible. The same is sometimes claimed about Rust.

In more permissive languages like Go, however, almost every parallel program is wrong in one way or another. For example, it is extremely common for parallel Go libraries to stop in a deadlock. The situation is so bad that I have stopped using concurrent Go libraries, except for the well-implemented standard libraries. Even then, many package authors don't tell you what they do internally and their libraries lock up in unexpected places. For example, I've recently been looking for a parallel tree walker library in Go, and every package I tried stopped with an infinite deadlock on my file system, presumably due to the interplay between error handling and spawning goroutines. Ironically, even many goroutine scheduling libraries I've tried have locked themselves up.

Re: Is parallel programming hard, and, if so, what can you do about it?

#36

Post by nurettin »

joe_the_user wrote: post #22
Koshkin wrote: post #3

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…

> Parallel programming is not hard per se I'm not sure how you can say that. Programming a fairly hard activity but the general consensus is adding parallelism to any given programming task, you make that task much, much harder. I don't think you find any version of parallel programming that isn't hard. The situation of parallelism, your commands happening in an unpredictable order, not synchronously, inherently, by…

>> I don't think you find any version of parallel programming that isn't hard.

Just serialize, pass data onto multiple processes, synchronize and wait on all of them to return their data back to main process. This is one kind of parallel programming that isn't hard.

But I cheatead and created expensive copies and processes and let the OS handle all the nightmare.

Re: Is parallel programming hard, and, if so, what can you do about it?

#37

Post by brazzy »

joe_the_user wrote: post #22
Koshkin wrote: post #3

Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its pe…

> Parallel programming is not hard per se I'm not sure how you can say that. Programming a fairly hard activity but the general consensus is adding parallelism to any given programming task, you make that task much, much harder. I don't think you find any version of parallel programming that isn't hard. The situation of parallelism, your commands happening in an unpredictable order, not synchronously, inherently, by…

> I don't think you find any version of parallel programming that isn't hard.

Using pure functional programming on an embarassingly parallel workload comes pretty close.

Re: Is parallel programming hard, and, if so, what can you do about it?

#38

Post by yvdriess »

Syzygies wrote: post #15

Haskell. I'm at almost fifty years of programming (APL was an early highlight; punched cards weren't) in many languages. Haskell isn't the be all / end all of languages, but when I want to add ten lines to a program and have it use every core, I don't know a better choice than Haskell. Nothing else comes close.

There is this halo around function programming languages being trivial to parallelize, but how they are implemented really matters. Historically, only few functional languages have been implemented with parallelism in mind (SISAL, pH).

The choice of the GHC Haskell to choose on-use evaluation as their non-strict evaluation strategy means that it cannot simply evaluate all function arguments in parallel. On-use means that if you do not use it it does not get evaluated, which some code relies on. Plus, as the trope goes, Haskell is implemented in a very imperative way, which can lead to some fun surprises when parallelizing aggressively.

There are like three different flavors of parallel programming in Haskell, differing in how much you want to trade off performance for having to know about compilation, memory management and execution details to make it run efficiently in non-trivial parallelism cases.

Re: Is parallel programming hard, and, if so, what can you do about it?

#39

Post by dnautics »

rubyn00bie wrote: post #24

Oh parallelism, the one thing I used to worry about more, and probably should (as far as using GPUs goes)... but also one I don't really sweat so much. Most of the reason, for me, was finding Elixir/Erlang AND their virtual machine BEAM. Until Elixir and BEAM for me, parallel programming had been difficult because the competing mental models in other languages pretty much make for a textbook fantasy. I'm sure some fo…

I, too, feel the same way about elixir (but my previous parallel programming experience was only in C). I just figured out how to safely dispatch long running NIFs using zig, so that should be fun!

> P.S. Anyone know if is there a language designed for running on GPUs designed to make it easier to facilitate programming even if they're slightly slower to execute, e.g. the Erlang/OTP/Beam of GPUs?

Try julia. I wish their gpu model was more actor-ish, but it's really easy to do GPU programming in it. You just take your existing code and re-type annotate it as GPU.

If I had a lot of time I would try to make a Julia/elixir bridge since I feel like the two languages will play nice with each other (Julia for computation and elixir for orchestration).

Re: Is parallel programming hard, and, if so, what can you do about it?

#40

Post by cerebellum42 »

clarry wrote: post #33
shaklee3 wrote: post #19

Earlier quoted context omitted.

Parallel programming is not hard if you follow the "just use a mutex and you avoid race conditions" mantra. However, this book dives deep into what mutexes actually do, what the hardware below is doing, and alternatives to a mutex for a speed increase. This can be anything from lockless algorithms to RCU, which the author of this book wrote it in Linux kernel. There's a comment above about Haskell and how you don't h…

> Parallel programming is not hard if you follow the "just use a mutex and you avoid race conditions" mantra. At one extreme of the spectrum, you can do everything under a big lock and be effectively single-threaded. At the other extreme, you can use lots of mutexes and fine grained locking and end up with a mess of potential deadlocks and races and "here be dragons" because the original programmer did not anticipate…

Sometimes it can be a nice way out to quickly lock the data, copy it and then unlock, if it's acceptable to have a task using outdated data occasionally. If that is not acceptable, there are Read-Write-Locks, which do guarantee that the data being used is current and allow for concurrent read access, but come with the drawback that write accesses might have to wait fairly long until the pointer can be fully locked.

What I'm saying is, don't limit yourself to Mutexes, there's a whole zoo of synchronization primitives and each of them has a specific use case that it's ideal for.

Recently I've dealt with a lot of synchronization in Rust, and I would say it's been as pleasant as it could be, considering the subject matter. Deadlocks are still possible, but at least the compiler stops you from accidentally unlocking a Mutex too early.

Post reply on HN