Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is parallel programming hard, and, if so, what can you do about it?

mirrors.edge.kernel.org

Post reply on HN
Print view View original

81–90 of 199 posts

Re: Is parallel programming hard, and, if so, what can you do about it?

#81

Post by [deleted] »

Retric wrote: post #71
bob1029 wrote: post #59

Earlier quoted context omitted.

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas. What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough. Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude.…

> What problems exist that generate events or commands faster than 500 million per second? AAA games, Google search, Weather simulation, etc? I mean it depends on what level of granularity you’re talking about, but many problems have a great deal going on under the hood and need to be multi threaded.

[deleted]

Re: Is parallel programming hard, and, if so, what can you do about it?

#82

Post by ascar »

samsquire wrote: post #16
ascar wrote: post #14

Earlier quoted context omitted.

> I am really interested in parallel, asynchronous, multithreading, coroutine, futures programming so it's what I spend my days thinking about and blogging about it. That's not parallel programming though. Parallell programming deals with the parallelization of a single sequential algorithm or program (e.g. weather simulation) across multiple threads, CPU or machines, usually with the requirement of real time synchro…

What would you call what I'm working on, it is "parallel" but probably doesn't come under "parallel programming" in the literature. I'm still interested in it though, just haven't got around to it yet, it's a large space. I'm not working on novel parallel algorithms that solve computer science problems, I am interested in parallelism as a general principle for coordinating systems activities. When I get time to delve…

What you're looking at is pretty much concurrent programming.

https://en.wikipedia.org/wiki/Concurrent_computing

Re: Is parallel programming hard, and, if so, what can you do about it?

#83

Post by bob1029 »

Retric wrote: post #71
bob1029 wrote: post #59

Earlier quoted context omitted.

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas. What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough. Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude.…

> What problems exist that generate events or commands faster than 500 million per second? AAA games, Google search, Weather simulation, etc? I mean it depends on what level of granularity you’re talking about, but many problems have a great deal going on under the hood and need to be multi threaded.

I would add a qualifier of "serializable" to those events or commands. This is the crux of why fintech goes this path. Every order affects subsequent orders and you must deal with everything in the exact sequence received.

The cases you noted are great examples of things that do justify going across the PCIe bus or to another datacenter.

Re: Is parallel programming hard, and, if so, what can you do about it?

#84

Post by ascar »

I see a lot of confusion between parallel programming [1] and concurrent programming [2] in the comments here.

The former and what this book is about deals with the problem of parallelizaing a single sequential program. There usually is strong interaction or dependencies between elements and progress needs synchronization. E.g. timestep iterations in real-time simulations that need synchronization with data communication after each timestep. These simulation also tend to get way to big to be run on a single machine, lest a single thread, and get scaled up to millions of cores/threads in supercomputers.

Concurrent programming is what most developers working with the internet are more familiar with. You have mostly independent tasks that you want to run concurrently. "A concurrent system is one where a computation can advance without waiting for all other computations to complete." [2] E.g. nginx serving thousands of user requests at the same time.

The problem domains have a lot of overlap on the basics (e.g. threading), however the focus is very different. Things like synchronization (mutex, barriers), cache locality and memory bandwith & latency play a central role in parallel programming, while concurrent programming focuses more on the engineering challenge of distributing independent tasks across multiple threads or machines.

[1] https://en.wikipedia.org/wiki/Parallel_computing

[2] https://en.wikipedia.org/wiki/Concurrent_computing

Re: Is parallel programming hard, and, if so, what can you do about it?

#85

Post by spacechild1 »

Retric wrote: post #71
bob1029 wrote: post #59

Earlier quoted context omitted.

Fintech has mostly determined that 1 thread can get the job done. See LMAX disruptor and related ideas. What problems exist that generate events or commands faster than 500 million per second? This is potentially the upper bar for 1 thread if you are clever enough. Latency is the real thing you want to get away from. Adding more than one CPU into the mix screws up the hottest possible path by ~2 orders of magnitude.…

> What problems exist that generate events or commands faster than 500 million per second? AAA games, Google search, Weather simulation, etc? I mean it depends on what level of granularity you’re talking about, but many problems have a great deal going on under the hood and need to be multi threaded.

Don't forget about audio!

Re: Is parallel programming hard, and, if so, what can you do about it?

#86

Post by Hbruz0 »

remontoire wrote: post #27

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

Link for that ?

Re: Is parallel programming hard, and, if so, what can you do about it?

#87

Post by nextaccountic »

aleph_minus_one wrote: post #39
vmfunction wrote: post #32

Earlier quoted context omitted.

what about "green threads" that is not managed by the OS like https://tokio.rs ?

Green threads do not make use of multiple cores of a modern processor.

M:N green threads run M green threads on N OS threads and thus use up to N processor cores

Re: Is parallel programming hard, and, if so, what can you do about it?

#88

Post by clueless »

Hbruz0 wrote: post #86
remontoire wrote: post #27

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

Link for that ?

https://www.youtube.com/watch?v=VgSQ1GOC86s

Re: Is parallel programming hard, and, if so, what can you do about it?

#89

Post by kprotty »

williamcotton wrote: post #36
Const-me wrote: post #24

As a developer, I often choose higher-level APIs not listed in that article. On Windows, OSX and iOS the OS userland already implements general, and relatively easy to use, thread pools. On Windows, see CreateThreadpoolWork, WaitForThreadpoolWorkCallbacks, etc. It’s easier to use threads with locks while someone else is managing these threads. On Apple, the pool is called “grand central dispatch” and does pretty much…

GCD/libdispatch is a fantastic approach to concurrency and you can build and install support for non-Apple operating systems: https://github.com/apple/swift-corelibs-libdispatch Here’s a simple echo server: https://github.com/williamcotton/c_playground/blob/master/sr... Here’s a simple multithreaded database pool: https://github.com/williamcotton/express-c/blob/master/src/d...

libdispatch idea of using specific queues for serialized concurrency is nice, but its abstractions on top are unfortunately not as efficiently designed as it could be; `dispatch_source` doesn't allow for direct completion based IO schemes (io_uring, IOCP), pushing to a `dispatch_queue` always requires a heap allocation, `dispatch_semaphore/dispatch_sync` blocks the thread instead of yielding asychronously (can cause "thread explosion"). Systems like Go don't have these constraints I don't think.

Re: Is parallel programming hard, and, if so, what can you do about it?

#90

Post by nradov »

remontoire wrote: post #27

Watching geohot code a general matrix multiply algorithm from 0.9 GFLOPS and optimising it to 100 glops by only tinkering with cache locality, it makes me wonder how much effort should be put into single threaded performance before ever thinking about multi threading

That appears to be one of the major design goals for the Mojo programming language. It allows the developer to code at a high level of abstraction and indicate where parallel execution should be used. Then the execution environment automatically optimizes that at runtime based on the actual hardware available. That hardware may change drastically over the full lifecycle of the application code so in the future it will automatically take advantage of hardware advances such as the introduction of new types of specialized co-processors.

https://www.modular.com/mojo

Post reply on HN