Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Think about Parallel Programming: Not! [video] (2021)

infoq.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: How to Think about Parallel Programming: Not! [video] (2021)

#11

Post by [deleted] »

bee_rider wrote: post #4
fifilura wrote: post #3

Disclaimer: at work so I didn't watch the video. For loops are the "goto":s of the parallel programming era. Ditch them and the rest can be handled by the programming language abstraction. Why? Because they 1. Enforce order of execution and 2. Allow breaking computation after a certain number of iterations.

I’ve always been surprised that we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations. It would be convenient for stuff like multi-core parallelism… and also for stuff like out of order execution! Not sure how “break” would be interpreted in this context. Maybe it should make the program crash, or it could be equivalent to “continue…

[deleted]

Re: How to Think about Parallel Programming: Not! [video] (2021)

#12

Post by dkarl »

epgui wrote: post #9
bee_rider wrote: post #4

Earlier quoted context omitted.

I’ve always been surprised that we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations. It would be convenient for stuff like multi-core parallelism… and also for stuff like out of order execution! Not sure how “break” would be interpreted in this context. Maybe it should make the program crash, or it could be equivalent to “continue…

> we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations Uhhh... we don't? It seems to me like we do. This is a solved problem. Depending on what you're trying to do, there's map, reduce, comprehensions, etc.

And for those who also don't want to be forced to sequence the computations, i.e., wanting to run them concurrently and potentially in parallel, each approach to concurrency supports its own version of this.

For example, choosing Scala on the JVM because that's what I know best, the language provides a rich set of maps, folds, etc., and the major libraries for different approaches to concurrency (futures, actors, effect systems) all provide ways to transform a collection of computations into a collection of concurrent operations.

Curious if the poster who said "we don't have a really widely supported construct" works in a language that lacks a rich concurrency ecosystem or if they want support baked into their language.

Re: How to Think about Parallel Programming: Not! [video] (2021)

#13

Post by dkarl »

vlovich123 wrote: post #10
bee_rider wrote: post #4

Earlier quoted context omitted.

I’ve always been surprised that we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations. It would be convenient for stuff like multi-core parallelism… and also for stuff like out of order execution! Not sure how “break” would be interpreted in this context. Maybe it should make the program crash, or it could be equivalent to “continue…

We actually do have the abstractions but the problem is that the vast majority of for loops don’t benefit - you need to have so much work that the overhead of coordinating the threads is sufficient. Additionally, you’ve got all sorts of secondary effects like cache write contention that will fight any win you try to extract out of for loops parallelism. What we’ve been learning for a long time as an industry is that…

Granted this probably isn't the parallel application that the other poster was envisioning, but it can be extremely useful when a computation depends on a large number of I/O-bound tasks that may fail, like when you are servicing a request with a high fan-out to other services, and you need to respond in a fixed time with the best information you have.

For example, if you need to respond to a request in 100ms and it depends on 100 service calls, you can make 100 calls with a 80ms timeout; get 90 quick responses, including two transient errors, and immediately retry the errors; get eight more successful responses and two timeouts; and then send the response within the SLA using the 98 responses you received.

Re: How to Think about Parallel Programming: Not! [video] (2021)

#14

Post by epgui »

dkarl wrote: post #12
epgui wrote: post #9

Earlier quoted context omitted.

> we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations Uhhh... we don't? It seems to me like we do. This is a solved problem. Depending on what you're trying to do, there's map, reduce, comprehensions, etc.

And for those who also don't want to be forced to sequence the computations, i.e., wanting to run them concurrently and potentially in parallel, each approach to concurrency supports its own version of this. For example, choosing Scala on the JVM because that's what I know best, the language provides a rich set of maps, folds, etc., and the major libraries for different approaches to concurrency (futures, actors, eff…

That's right-- Personally I like to think in functional programming terms, and with FP concurrency/parallelism is more or less a non-issue.

Re: How to Think about Parallel Programming: Not! [video] (2021)

#15

Post by mystified5016 »

bee_rider wrote: post #4
fifilura wrote: post #3

Disclaimer: at work so I didn't watch the video. For loops are the "goto":s of the parallel programming era. Ditch them and the rest can be handled by the programming language abstraction. Why? Because they 1. Enforce order of execution and 2. Allow breaking computation after a certain number of iterations.

I’ve always been surprised that we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations. It would be convenient for stuff like multi-core parallelism… and also for stuff like out of order execution! Not sure how “break” would be interpreted in this context. Maybe it should make the program crash, or it could be equivalent to “continue…

> like a for loop, but with no dependency allowed between iterations

"Break" is a dependency between iterations, and really only makes sense in a sequential iteration. In a parallel for loop, you can break from the current iteration, but the next is probably already running.

If you want any iteration to be able to cancel all others, they have to be linked somehow. Giving every task a shared cancellation token might be simplest. Or you turn your for loop into a sort of task pool that intelligently herds threads in the background and can consume and relay cancellation requests.

But I agree, we need a new paradigm for parallel programming. For loops just don't cut it, despite being one of the most natrual-feeling programming concepts.

C#'s Parallel.For and ForEach are a step in the right direction, but very unergonomic and unintuitive. I think we could get by with just bolting parallelism onto for loops, but we need a fundamentally parallel concept. I assume it'd look something like cuda programming but I really don't know.

Re: How to Think about Parallel Programming: Not! [video] (2021)

#16

Post by bee_rider »

two_handfuls wrote: post #7
bee_rider wrote: post #4

Earlier quoted context omitted.

I’ve always been surprised that we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations. It would be convenient for stuff like multi-core parallelism… and also for stuff like out of order execution! Not sure how “break” would be interpreted in this context. Maybe it should make the program crash, or it could be equivalent to “continue…

They're not in the language proper, but "parallel for" is a common construct. I've seen it in C# and Rust, but I'm sure other languages have it too. It may be a good idea to use a framework with explicitly stateless "tasks" and an orchestrator (parallel, distributed, or both). This is what Spark, Tensorflow, Beam and others do. Those will have a "parallel for" as well, but now in addition to threads you can use remot…

The big C and Fortran compilers have openMP support, which includes parallel for loops. They just feel kind of… bolted on, being a pragma based language extension. And what I really want to express to the thing isn’t “fork here” but “here are some independent operations, tell the optimizing compiler about it,” and then the optimizing compiler can (among other transformations also decide to sprinkle some threads in there)

Re: How to Think about Parallel Programming: Not! [video] (2021)

#17

Post by fifilura »

mystified5016 wrote: post #15
bee_rider wrote: post #4

Earlier quoted context omitted.

I’ve always been surprised that we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations. It would be convenient for stuff like multi-core parallelism… and also for stuff like out of order execution! Not sure how “break” would be interpreted in this context. Maybe it should make the program crash, or it could be equivalent to “continue…

> like a for loop, but with no dependency allowed between iterations "Break" is a dependency between iterations, and really only makes sense in a sequential iteration. In a parallel for loop, you can break from the current iteration, but the next is probably already running. If you want any iteration to be able to cancel all others, they have to be linked somehow. Giving every task a shared cancellation token might b…

I believe this lecture is universal för both SQL and CUDA.

https://gfxcourses.stanford.edu/cs149/fall24/lecture/datapar...

Re: How to Think about Parallel Programming: Not! [video] (2021)

#18

Post by vlovich123 »

dkarl wrote: post #13
vlovich123 wrote: post #10

Earlier quoted context omitted.

We actually do have the abstractions but the problem is that the vast majority of for loops don’t benefit - you need to have so much work that the overhead of coordinating the threads is sufficient. Additionally, you’ve got all sorts of secondary effects like cache write contention that will fight any win you try to extract out of for loops parallelism. What we’ve been learning for a long time as an industry is that…

Granted this probably isn't the parallel application that the other poster was envisioning, but it can be extremely useful when a computation depends on a large number of I/O-bound tasks that may fail, like when you are servicing a request with a high fan-out to other services, and you need to respond in a fixed time with the best information you have. For example, if you need to respond to a request in 100ms and it…

That doesn't require parallelism, just concurrency. But yes, you'd use a similar task-local map/reduce construct to express doing a bunch of concurrent I/O in parallel (spawning each I/O on a separate thread would be counter-productive & a hack to enable not adding an event loop / async I/O).

Re: How to Think about Parallel Programming: Not! [video] (2021)

#19

Post by fifilura »

vlovich123 wrote: post #10
bee_rider wrote: post #4

Earlier quoted context omitted.

I’ve always been surprised that we don’t have a really widely supported construct in programming that is like a for loop, but with no dependency allowed between iterations. It would be convenient for stuff like multi-core parallelism… and also for stuff like out of order execution! Not sure how “break” would be interpreted in this context. Maybe it should make the program crash, or it could be equivalent to “continue…

We actually do have the abstractions but the problem is that the vast majority of for loops don’t benefit - you need to have so much work that the overhead of coordinating the threads is sufficient. Additionally, you’ve got all sorts of secondary effects like cache write contention that will fight any win you try to extract out of for loops parallelism. What we’ve been learning for a long time as an industry is that…

Let the engine or compiler decide if it is small enought to run on one core.

Re: How to Think about Parallel Programming: Not! [video] (2021)

#20

Post by vlovich123 »

fifilura wrote: post #19
vlovich123 wrote: post #10

Earlier quoted context omitted.

We actually do have the abstractions but the problem is that the vast majority of for loops don’t benefit - you need to have so much work that the overhead of coordinating the threads is sufficient. Additionally, you’ve got all sorts of secondary effects like cache write contention that will fight any win you try to extract out of for loops parallelism. What we’ve been learning for a long time as an industry is that…

Let the engine or compiler decide if it is small enought to run on one core.

That would be great if the engine or compiler had that kind of capability but building that requires solving the halting problem.

Even if you try to do it with heuristics, go ask Itanium how that worked out for them and they tried a much simpler problem than what you’re proposing.

Post reply on HN