Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

mirrors.edge.kernel.org

91–100 of 199 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#91
post #72

On scientific computing (computational fluid dynamics, computational electromagnetics, etc.), parallel programing is a must. Most of the algorithms are not embarrassingly parallel, and need a significant amount of communication between threads during runtime. We mostly use one of the many MPI [0] libraries available for desktop and high-performance computing machines. Using these programming paradigms is difficult bu…

So this might be a naive question, but is it literally that you're simulating a fluid in parallel by having each thread simulate a portion of the space? And then the message passing between threads is the data about the fluid on the boundary?

Re: Is parallel programming hard, and, if so, what can you do about it?

#92
post #47

The symtoms that lead up to me learning about the need for locks in multi threaded servers was painful. That the state could change from one line of code to the next line, how could it change when I just set it a micro second ago? Because the user clicked the button twice and the call was executed simultaneously but in different threads because my new server had many cores. Then I learned about async programming, but…

> the state could change from one line of code to the next line

> I no longer remember what was so difficult about it

You or (your libraries) probably stopped mutating things. No mutation means you can rely on values you used several lines ago.

This is the perennial HN discussion whenever the word 'functional' comes up. "But x=x+1 is simple to teach to beginners" is the rallying cry. Well, this is why not.

Also, > the need for locks

This goes away if you don't mutate.

Re: Is parallel programming hard, and, if so, what can you do about it?

#93
post #59

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas. What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough. Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude.…

As far as I know the LMAX disrupter is a kind of queue/buffer to send data from one thread/task to another.

Typically, some of the tasks run on different cores. The LMAX disruptor is designed such that there is no huge delay due to cache coherency. It is slow to sync the cache of one core to the cache of another core when both cores write to the same address in RAM. The LMAX disruptor is designed that each memory location is (mostly) written to by at most thread/core.

How is the LMAX disrupter relevant for programs with 1 core?

Re: Is parallel programming hard, and, if so, what can you do about it?

#94
post #84

I see a lot of confusion between parallel programming [1] and concurrent programming [2] in the comments here. The former and what this book is about deals with the problem of parallelizaing a single sequential program. There usually is strong interaction or dependencies between elements and progress needs synchronization. E.g. timestep iterations in real-time simulations that need synchronization with data communica…

The book covers it in Appendix A.6 (p 424) in the v2023.06.11a PDF file.

> A.6 What is the Difference Between “Concurrent” and “Parallel”?

> From a classic computing perspective, “concurrent” and “parallel” are clearly synonyms. However, this has not stopped many people from drawing distinctions between the two, and it turns out that these distinctions can be understood from a couple of different perspectives.

> The first perspective treats “parallel” as an abbreviation for “data parallel”, and treats “concurrent” as pretty much everything else. From this perspective, in parallel computing, each partition of the overall problem can proceed completely independently, with no communication with other partitions. In this case, little or no coordination among partitions is required. In contrast, concurrent computing might well have tight interdependencies, in the form of contended locks, transactions, or other synchronization mechanisms.

> This of course begs the question of why such a distinction matters, which brings us to the second perspective, that of the underlying scheduler. Schedulers come in a wide range of complexities and capabilities, and as a rough rule of thumb, the more tightly and irregularly a set oparallel processes communicate, the higher the level of sophistication required from the scheduler. As such, parallel computing’s avoidance of interdependencies means that parallel-computing programs run well on the least-capable schedulers. In fact, a pure parallel-computing program can run successfully after being arbitrarily subdivided and interleaved onto a uniprocessor. In contrast, concurrent computing programs might well require extreme subtlety on the part of the scheduler.

Re: Is parallel programming hard, and, if so, what can you do about it?

#95
post #72

On scientific computing (computational fluid dynamics, computational electromagnetics, etc.), parallel programing is a must. Most of the algorithms are not embarrassingly parallel, and need a significant amount of communication between threads during runtime. We mostly use one of the many MPI [0] libraries available for desktop and high-performance computing machines. Using these programming paradigms is difficult bu…

So this might be a naive question, but is it literally that you're simulating a fluid in parallel by having each thread simulate a portion of the space? And then the message passing between threads is the data about the fluid on the boundary?

Computational fluid dynamics is actually a problem where parallelization doesn't get you much. It is primarily limited by memory bandwidth.

Re: Is parallel programming hard, and, if so, what can you do about it?

#96
post #59

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas. What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough. Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude.…

> God forbid you have to wait on the GPU or network. If you have to talk to those targets, it had better be worth the trip.

few programs are CPU-bound, most programs are bottlenecked on I/O waits like these

Re: Is parallel programming hard, and, if so, what can you do about it?

#97
post #50

Earlier quoted context omitted.

Correct me if I’m wrong, but isn’t concurrency enough for “most” use-cases? When does one really need true parallelism?

Concurrency is typically a great solution of IO bound tasks, which is why it figures so prominently in modern day webdev. It's also essential for any UIs in which case you don't want a single threaded process to be blocking user interaction just because it's running a task that takes time to complete. Parallelism is used to solve CPU bound problems. Obvious examples are things like efficient matrix multiplication. Ho…

I don't think IO bound and CPU bound problems are the correct way of differentiating between the two. Parallel programming deals with real-time problems with data dependencies and synchronization requirements. While concurrent programming deals with independent computations, where progress or completion of one does not depend on the progress of another.

I agree however they do ultimateively solve two very different domains of problems and there is a lot of confusion going on here.

On a side note: matrix multiplcation is generally not a CPU bound problem, but a memory bound one.

Re: Is parallel programming hard, and, if so, what can you do about it?

#99
post #94
post #84

I see a lot of confusion between parallel programming [1] and concurrent programming [2] in the comments here. The former and what this book is about deals with the problem of parallelizaing a single sequential program. There usually is strong interaction or dependencies between elements and progress needs synchronization. E.g. timestep iterations in real-time simulations that need synchronization with data communica…

The book covers it in Appendix A.6 (p 424) in the v2023.06.11a PDF file. > A.6 What is the Difference Between “Concurrent” and “Parallel”? > From a classic computing perspective, “concurrent” and “parallel” are clearly synonyms. However, this has not stopped many people from drawing distinctions between the two, and it turns out that these distinctions can be understood from a couple of different perspectives. > The…

that description is... not accurate

concurrent is about logical independence, parallel is about physical independence

Re: Is parallel programming hard, and, if so, what can you do about it?

#100
post #59

Earlier quoted context omitted.

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas. What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough. Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude.…

As far as I know the LMAX disrupter is a kind of queue/buffer to send data from one thread/task to another. Typically, some of the tasks run on different cores. The LMAX disruptor is designed such that there is no huge delay due to cache coherency. It is slow to sync the cache of one core to the cache of another core when both cores write to the same address in RAM. The LMAX disruptor is designed that each memory loc…

> How is the LMAX disrupter relevant for programs with 1 core?

It is not relevant outside the problem area of needing to communicate between threads. The #1 case I use it for is MPSC where I have something like an AspNetCore/TCP frontend and a custom database / event processor / rules engine that it needs to talk to.

Post reply on HN