Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

mirrors.edge.kernel.org

Post reply on HN
Print view View original

51–60 of 199 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#51

Post by CyberDildonics »

bullen wrote: post #34

Only 2 ways into parallelism: - Arrays of 64 byte Structures with C. (client) - Java. (server) Everything else is not atomic.

This makes no sense.

- Arrays of 64 byte Structures with C. (client)

All you need for parallelism is to spawn threads and give each thread a range of an array to work on. You don't need each element to be atomic because you aren't synchronizing on each element.

What makes you think you need specific data and a specific language?

- Java. (server)

This makes even less sense. A language doesn't matter and how it's used doesn't matter.

Where did you get this idea? Can you link something?

Everything else is not atomic.

I think you don't understand what parallelism means. Read about openMP, it is very simple.

Re: Is parallel programming hard, and, if so, what can you do about it?

#52

Post by adsfgiodsnrio »

bee_rider wrote: post #28
cj wrote: post #23

This reminds me when I was going through the YC accelerator in 2012. We were building a web-based email client, and PG didn’t like the idea. He pulled our team aside during one of the batch-wide Tues night dinners and suggested we pivot to building something that could take single threaded programs and quickly/easily make them multi-threaded. No one on our team knew anything about threading (none of us had even gradu…

If you had something in 2012 that could magically convert a single threaded program to a parallel one, it would be pretty useful. Seems incredibly difficult to build, though.

Do you think a Philosopher's Stone might have some value, too? I think Y Combinator should look into it. Maybe an Elixir of Immortality?

An automatically parallelizing compiler that works in general is one of the holy grails of computer science. They have been extensively researched for decades and we have little to show for it. If anyone managed to make one it would instantly render the entire computer industry obsolete.

We have systems that work in specialized cases with a whole lot of manual intervention. Even then, the executable is often less efficient than the serial version.

Re: Is parallel programming hard, and, if so, what can you do about it?

#53

Post by twoodfin »

cmrdporcupine wrote: post #40

Concurrent state management is hard, and I remain disappointed that software transactional memory -- and pushing a DB-style approach to non-DB workloads generally -- hasn't caught on. I think most "application" type programs would benefit from being able to manage state in terms of higher level transactional operations and let their runtime take care of serializing and avoiding deadlock and race conditions. Developer…

You might be interested in Joe Duffy’s retrospective on Microsoft’s failed experiment to integrate STM into .NET:

https://joeduffyblog.com/2010/01/03/a-brief-retrospective-on...

Re: Is parallel programming hard, and, if so, what can you do about it?

#54

Post by jefc1111 »

PheonixPharts wrote: post #31
jefc1111 wrote: post #19

My answers to the two questions in the post title: "Yes" and "Read Concurrent Data Processing in Elixir [1]" I have dealt with concurrency in PHP (good times with Laravel Horizon) in a couple of different ways and in NodeJS (never feels good, to me). The BEAM has some great primitives for concurrency and the book mentioned above walks the reader through them at a good pace. I read it cover to cover and it really enri…

Concurrency is not the same as parallelism. You can have single threaded concurrent programs.

Oh yes, good call! Still, was a good book...

Re: Is parallel programming hard, and, if so, what can you do about it?

#55

Post by bsdetector »

tm-guimaraes wrote: post #13
flohofwoe wrote: post #10

Earlier quoted context omitted.

It's only easy if the items that are iterated over are entirely self contained, and the code in the loop body also does not need (write-)access to any other state that might be shared between threads. Rust has such limitations built into the language, but this is also one of the points that make Rust "difficult" (because you actually need to think first about how you organize your data to be "multi-threading-compatib…

That’s only safety. It still does not guarantee that performance would improve.

A built-in parallel foreach can't guarantee better performance because the hardware isn't there. The parallel part is scheduled on normal threads by the kernel and this is too unpredictable.

But if each OS-scheduled CPU was really a cluster of for instance 1 fast + 64 slow cores then it could have special instructions to split a low-level loop on these slow cores and to facilitate blocking and rejoining back into a single stream. A 'parallel branch' instruction could take index, limit, and estimated instruction count and the CPU itself could decide whether to run it in a single fast stream or in parallel on slow cores.

There's some technical challenges like the size of a register file (so probably uninterruptible), but this may be where we're going since there's diminishing returns from making a single CPU faster and there's lots of opportunity for parallel processing that GPUs are too clumsy for.

Re: Is parallel programming hard, and, if so, what can you do about it?

#56

Post by shwestrick »

cpach wrote: post #50
PheonixPharts wrote: post #30

Earlier quoted context omitted.

There are multiple replies like this one, but it's a bit shocking to see that on Hacker News people don't know the difference between concurrent programming and parallel programming. Concurrency means that you can have multiple tasks running in the same time period, Parallelism means you have multiple tasks running at the same time . The most obvious demonstration of this is that you can (and many languages do) have…

Correct me if I’m wrong, but isn’t concurrency enough for “most” use-cases? When does one really need true parallelism?

Parallelism is only about performance, that's it. If you need something to go faster, parallelism is an option.

Looking into the future, parallelism is one of the only remaining techniques for scaling classical computing. Processors have stopped getting faster. Instead, they're getting fatter (more cores).

Re: Is parallel programming hard, and, if so, what can you do about it?

#57

Post by jchanimal »

klabb3 wrote: post #21
ascar wrote: post #15

Earlier quoted context omitted.

> concurrent programming If you're interested in concurrent programming this book won't give you much. The topics focus on the parallelization of a sequential algorithm and go into detail about things like synchronization (locking, barriers), important HW details (like caches and CPU pipelining) and algorithmic approaches. Imagine a weather simulation and not concurrent requests to a web server.

Good to know! Yes I’m less interested in “the art of programming but parallel” (partly because I don’t have PhD level ambitions) and more interested in how to effectively program day-to-day boring stuff that also needs concurrency. My assessment is that the latter is nowhere near “solved” (hesitant to use that word because craft unlike eg proofs is about trade offs), in the sense that concurrency differs a lot across…

Turing equivalence says that all logic can be translated to other paradigms. But thread interactions are not logic they are based in real time. So the differences in how they treat time will show up as differences in the implementation. I guess you could create a worst of all of the above polymorphic abstraction that looks the same no matter which way you’re using it, but I don’t know if we would benefit much from that.

Re: Is parallel programming hard, and, if so, what can you do about it?

#58

Post by bee_rider »

adsfgiodsnrio wrote: post #52
bee_rider wrote: post #28

Earlier quoted context omitted.

If you had something in 2012 that could magically convert a single threaded program to a parallel one, it would be pretty useful. Seems incredibly difficult to build, though.

Do you think a Philosopher's Stone might have some value, too? I think Y Combinator should look into it. Maybe an Elixir of Immortality? An automatically parallelizing compiler that works in general is one of the holy grails of computer science. They have been extensively researched for decades and we have little to show for it. If anyone managed to make one it would instantly render the entire computer industry obso…

I also think it is an unrealistic suggestion, in case that didn’t come through in my first comment! Haha.

But I’m sure the hoped-for result was just another one of those specialized tools, specialized for some Y Combinator niche. Which is also probably an unrealistic idea for a team that isn’t interested in that sort of thing, but at least isn’t totally ridiculously stupid.

Re: Is parallel programming hard, and, if so, what can you do about it?

#59

Post by bob1029 »

remontoire wrote: post #27

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas.

What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough.

Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude. God forbid you have to wait on the GPU or network. If you have to talk to those targets, it had better be worth the trip.

Re: Is parallel programming hard, and, if so, what can you do about it?

#60

Post by DayDollar »

Ah, the old debate and approach. Centralize the control, Decentralize problem handling, centralize the knowledge needed for untangling parallelism, realize you just created a centralized authority, undoing the parallelism.

Then there is the physics engine view to all things. Data is just sand washing against the shoreline of parallel computation instances and interaction, is just sticking these dataparticles together, taking them out of the general flow to be interacted on in unison, carrying the resolution authority within them as long as they are "clumbed" together. Data is just particles, the programs interacting on it are just a limited number of program cells, traversing it and interaction clumps it together, making the processing of it slower, but also centralized to computation node. A centralized node, processing a huge clump of interacting data, even throws a sort of "relativity" shadow, as other, smaller, non interacting data, is processed faster and leaping ahead in the interactions.

Post reply on HN