Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

mirrors.edge.kernel.org

Post reply on HN
Print view View original

161–170 of 199 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#161

Post by pshc »

imtringued wrote: post #148
taeric wrote: post #126

Earlier quoted context omitted.

Thinking of graphics, though, i would assume most of that is in the GPU side. Simulations do make sense, but I see games like Factorio still focused on single thread first. And then look for natural parallel segments. That is all to say that millions of events still feels like a lot. I am not shocked to know it can and does happen.

There are no good solutions for something like factorio. There are solutions that work but they aren't worth the trouble. My personal recommendation is that you split the world into independent chunks. A big interconnected factorio map is a nightmare scenario because there is hardly anywhere where you can neatly split things up. Just one conveyor belt and you lose. Aka parallelize disconnected subgraphs. So the game…

Just dreaming over here, but if someone had the opportunity to rebuild a Factorio from the ground up, I bet they could design something massively scalable. Something based in cell automata, like how water flow works in Minecraft. Current cell state = f(prev cell state, neighboring cell states).

It would take some careful work to ensure that items didn't get duplicated or lost at junctions, and a back-pressure system for conveyor belt queues. Electrical signals would be transmitted at some speed limit.

Re: Is parallel programming hard, and, if so, what can you do about it?

#162

Post by CyberDildonics »

vore wrote: post #158
CyberDildonics wrote: post #151

Earlier quoted context omitted.

Parallelism is a physical thing, concurrency is a logical thing. Fundamentally the difficulty is all about synchronization. People can try to split hairs and say there are two terms for two different things but ultimately it doesn't matter because the underlying problem is the same.

Single core concurrency doesn't have to deal with hardware memory synchronization.

What point are you trying to make now? You just said:

Parallelism is a physical thing, concurrency is a logical thing.

So by your own definition, wouldn't multi-core concurrency be parallelism?

Re: Is parallel programming hard, and, if so, what can you do about it?

#163

Post by Jtsummers »

vore wrote: post #158
CyberDildonics wrote: post #151

Earlier quoted context omitted.

Parallelism is a physical thing, concurrency is a logical thing. Fundamentally the difficulty is all about synchronization. People can try to split hairs and say there are two terms for two different things but ultimately it doesn't matter because the underlying problem is the same.

Single core concurrency doesn't have to deal with hardware memory synchronization.

> Single core concurrency doesn't have to deal with hardware memory synchronization.

Yes it does. If you have two threads which (for whatever reason) are sharing memory and they can be run in an arbitrary order, you have to deal with memory synchronization. If it's a single core machine, you still need to synchronize your memory access. Mutexes and semaphores predate systems with multiple processors.

Re: Is parallel programming hard, and, if so, what can you do about it?

#164

Post by z3t4 »

mrkeen wrote: post #92
z3t4 wrote: post #47

The symtoms that lead up to me learning about the need for locks in multi threaded servers was painful. That the state could change from one line of code to the next line, how could it change when I just set it a micro second ago? Because the user clicked the button twice and the call was executed simultaneously but in different threads because my new server had many cores. Then I learned about async programming, but…

> the state could change from one line of code to the next line > I no longer remember what was so difficult about it You or (your libraries) probably stopped mutating things. No mutation means you can rely on values you used several lines ago. This is the perennial HN discussion whenever the word 'functional' comes up. "But x=x+1 is simple to teach to beginners" is the rallying cry. Well, this is why not. Also, > th…

I like to philosophize about the universe, how at every state change a new universe is created. Like if the version of you I see is someone else, almost identical, and the version I interacted with have branched off into a new universe.

Re: Is parallel programming hard, and, if so, what can you do about it?

#165

Post by wizofaus »

preseinger wrote: post #153
crabbone wrote: post #141

Earlier quoted context omitted.

You have just repeated the nonsense I was talking about. The claim you repeat is meaningless. A program is either parallel / concurrent or not. The situation you describe (when there's a single processor core) isn't parallel or concurrent. In some sense, it emulates concurrent / parallel execution because it imitates the unpredictable ordering of code execution, which sure has its uses... but the whole point of deali…

a program (as written) is either concurrent or not, a program (as executed) is either parallel or not a program which is not written as concurrent can never be executed as parallel a program which is written as concurrent can be executed as parallel, or not > The situation you describe (when there's a single processor core) isn't parallel or concurrent. a program running on a single core can never be parallel, but it…

"a program which is not written as concurrent can never be executed as parallel"

Unless of course you're running multiple independent instances of it, each with different parameters, which I gather is a pretty common way of running long running CPU- intensive operations on large data sets. Presumably there's often some final separate step that may be needed to combine the results once they're all finished, which if run manually obviates any concurrency concerns at the software level.

Re: Is parallel programming hard, and, if so, what can you do about it?

#166

Post by preseinger »

wizofaus wrote: post #165
preseinger wrote: post #153

Earlier quoted context omitted.

a program (as written) is either concurrent or not, a program (as executed) is either parallel or not a program which is not written as concurrent can never be executed as parallel a program which is written as concurrent can be executed as parallel, or not > The situation you describe (when there's a single processor core) isn't parallel or concurrent. a program running on a single core can never be parallel, but it…

"a program which is not written as concurrent can never be executed as parallel" Unless of course you're running multiple independent instances of it, each with different parameters, which I gather is a pretty common way of running long running CPU- intensive operations on large data sets. Presumably there's often some final separate step that may be needed to combine the results once they're all finished, which if r…

in which case that program is being executed N times, so none of this applies

("a program [as executed]" is a single instance (process) of a binary)

Re: Is parallel programming hard, and, if so, what can you do about it?

#167

Post by uguuo_o »

elil17 wrote: post #95
OscarCunningham wrote: post #91

Earlier quoted context omitted.

So this might be a naive question, but is it literally that you're simulating a fluid in parallel by having each thread simulate a portion of the space? And then the message passing between threads is the data about the fluid on the boundary?

Computational fluid dynamics is actually a problem where parallelization doesn't get you much. It is primarily limited by memory bandwidth.

I am not sure where you get this information. Parallelization is everything in this space, hence why we have highly interconnected supercomputers to model the most difficult engineering problems. Typical runs use 30k+ cores for a single problem for weeks on end [0]. There are some special cases, such as Boltzmann/dvm [1] where invididual partitions of cells have millions of degrees of freedom, where memory bandwidth is the primary concerns. Even then, doing domain decomposition to a larger number of cores takes care of the issue.

[0] https://www.nas.nasa.gov/SC22/research/project12.html

[1] https://www.sciencedirect.com/science/article/abs/pii/S00219...

Re: Is parallel programming hard, and, if so, what can you do about it?

#168

Post by uguuo_o »

OscarCunningham wrote: post #91
uguuo_o wrote: post #72

On scientific computing (computational fluid dynamics, computational electromagnetics, etc.), parallel programing is a must. Most of the algorithms are not embarrassingly parallel, and need a significant amount of communication between threads during runtime. We mostly use one of the many MPI [0] libraries available for desktop and high-performance computing machines. Using these programming paradigms is difficult bu…

So this might be a naive question, but is it literally that you're simulating a fluid in parallel by having each thread simulate a portion of the space? And then the message passing between threads is the data about the fluid on the boundary?

Depending on the method, but generally yes, the physical domain is decomposed into the number of cores. Data is synchronized at processor boundaries at a very high rate to maintain consistency.

Re: Is parallel programming hard, and, if so, what can you do about it?

#169

Post by codys »

Const-me wrote: post #24

As a developer, I often choose higher-level APIs not listed in that article. On Windows, OSX and iOS the OS userland already implements general, and relatively easy to use, thread pools. On Windows, see CreateThreadpoolWork, WaitForThreadpoolWorkCallbacks, etc. It’s easier to use threads with locks while someone else is managing these threads. On Apple, the pool is called “grand central dispatch” and does pretty much…

Please don't use posix message queues (`mq_*`, `mq_overview`, etc) when you're writing a program with a single address space (like something that uses threads in a single process).

Posix message queues (the `mq_*` functions) are much slower than optimized shared memory queues using typical atomics and have semantics that are unexpected to most (they persist like files after process termination, because of what they are designed to be used for, they have system level limits on size and size of items, etc).

A simple benchmark vs rust's `std::sync::mpsc` queue shows `std::sync::mpsc` is 28.6 times faster when using 1 producer and 1 consumer, and is 37.32 times faster with 2 producers and 1 consumer.

Re: Is parallel programming hard, and, if so, what can you do about it?

#170

Post by kylewatson »

vore wrote: post #146
crabbone wrote: post #140

Earlier quoted context omitted.

1. That's not part of the language, it's part of an implementation. I.e. it means that the same language can be both statically-typed and not (according to your "definition"). Which is literally nonsense (in the sense true = false). 2. If something is commonly accepted doesn't make it anymore true. In the context of programming, a lot of commonly accepted beliefs are nonsense, this one isn't an exception. It's not ab…

What are you talking about? Languages are pretty explicitly designed to be statically- or dynamically-typed: e.g. take Python which relies a lot on having a dynamic typing discipline. Yes, you have Mypy and pytype, but those are pretty much different dialects. Also, what reputable sources are you even talking about? That is also such an unwarranted personal attack you've attached as well.

I think there is a point here if you ignore the grumpy old man delivery.

For example, even python which is dynamically typed, is strongly typed also, so you now have to not just know the distinction between statically typed and dynamically typed but also between strong and weak. Then stray away slightly from vanilla python interpreter/runtime to any of the other flavors and you now have to reason about compile time, runtime, interpretation time, bytecode, interop with jvm etc. So there is a point that the language is a way of expressing something and the implementation is where the devilish details lie. You can argue jython isn't python or whatever and that's all well and good, but you can't really discuss all of this with other reasonable humans without getting into the details of implementation. Sure, for a leetcode level of understanding it doesn't matter much, but try to do something sufficiently complicated like build an os extension in python that interops with your c++ based api and you'll have to think about the implementation of the projections, and then port it to arm and you'll have to think about it all over again.

Post reply on HN