Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

351–360 of 366 posts

Re: A viable solution for Python concurrency

#351

Post by globular-toast »

adwn wrote: post #310
globular-toast wrote: post #301

Earlier quoted context omitted.

The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.

> that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.

Oh come on. It should be obvious that I'm talking about the processes that can be split up in that way. Those problems are so common that someone literally wrote GNU parallel to solve them.

Re: A viable solution for Python concurrency

#352

Post by MainJane »

adwn wrote: post #310
globular-toast wrote: post #301

Earlier quoted context omitted.

The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.

> that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.

> Try using GNU parallel to sort a list of numbers,

`parsort` is part of GNU Parallel.

Re: A viable solution for Python concurrency

#353

Post by [deleted] »

MaXtreeM wrote: post #303

Slight off topic but I am curious about using bits in an integer for flags. As the article mentions Gross uses 2 least significat bits for flags and the rest is an integer for reference counting. When someone considers whether to use most significant bits or least significant bits are there any major differences? Is is easier to implement or faster because of processor architectures/instruction sets to use least sign…

[deleted]

Re: A viable solution for Python concurrency

#354

Post by detaro »

rfoo wrote: post #346
detaro wrote: post #345

Earlier quoted context omitted.

Why would webdevs care about work trying to remove the GIL?

Previous GIL removal attempts hurts single thread performance and it isn't that scalable, so people are usually by default dismissive. Most of Python codes depend on subtle details of CPython internal. For example sometimes it is just convenient to assume GIL exists (i.e. simplifies concurrency codes because "you know there are at most one thread running").

I haven't seen any appetite to even consider solutions that break the promises of the GIL and make currently atomic things non-atomic, so that second argument seems weird.

Re: A viable solution for Python concurrency

#355

Post by adwn »

globular-toast wrote: post #351
adwn wrote: post #310

Earlier quoted context omitted.

> that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.

Oh come on. It should be obvious that I'm talking about the processes that can be split up in that way. Those problems are so common that someone literally wrote GNU parallel to solve them.

> I'm talking about the processes that can be split up in that way

No, you weren't. You said: "[...] GNU parallel [...] works for any non-parallel process" (emphasis mine)

> Those problems are so common that someone literally wrote GNU parallel to solve them.

As part of my job I'm writing multi-threaded, parallel programs all the time, and in those years only a single problem would have been feasible to parallelize with GNU parallel; but since I was using Rust, it was trivial to do the parallelization right there in my code without having to resort an outer script/binary that calls GNU parallel on my program.

Re: A viable solution for Python concurrency

#356

Post by the__alchemist »

rfoo wrote: post #344
the__alchemist wrote: post #255

Maybe we just accept that Python isn't suitable for concurrency. There's a community of Python developers who don't want to branch out; write everything in Python and never learn or consider another language. Let them be. Let Python excel at its core competencies; use the right tool for the job.

> Let Python excel at its core competencies Python is notably very popular in two communities: web developer and scientific computing. The former usually yell loudly every time someone propose to remove GIL. Meanwhile everyone in the scientific computing community had to learn how to workaround GIL which absolutely sucks and sometimes just impossible. (e.g. I have a mostly memory-bandwidth-bound data loading pipeline…

Numpy/Scipy are great for prototyping, but IMO code the performant / concurrent stuff in C/C++/Rust/Julia once your prototype is done. Use the right tool for the job instead of trying to bend the wrong one.

Re: A viable solution for Python concurrency

#357

Post by rattray »

otterley wrote: post #325
rattray wrote: post #323

Earlier quoted context omitted.

The question at hand is concurrency, not single-threaded performance.

You said "high throughput" though. Your words, not mine :-)

Ah, I see the misconception - "high throughput" means a system processes a lot of stuff in a given span of time, not necessarily that each individual item is processed quickly. But many people use it like the latter.

Re: A viable solution for Python concurrency

#358

Post by adwn »

MainJane wrote: post #352
adwn wrote: post #310

Earlier quoted context omitted.

> that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.

> Try using GNU parallel to sort a list of numbers, `parsort` is part of GNU Parallel.

... and it uses a manually implemented post-processing step. You can't just run the sort program with GNU parallel and expect to get a fully sorted list.

Re: A viable solution for Python concurrency

#359

Post by MainJane »

adwn wrote: post #358
MainJane wrote: post #352

Earlier quoted context omitted.

> Try using GNU parallel to sort a list of numbers, `parsort` is part of GNU Parallel.

... and it uses a manually implemented post-processing step. You can't just run the sort program with GNU parallel and expect to get a fully sorted list.

> Try using GNU parallel to sort a list of numbers, [...] – it's not possible,

Yet it clearly is possible, so your blanket statement is clearly wrong.

`parsort` a simple wrapper, and this really goes for many uses of GNU Parallel: You need to prepare your data for the parallel step and post-process the output.

Maybe you originally meant to say: "Only for those processes, where you can preprocess the input and post-process the outputs."

Re: A viable solution for Python concurrency

#360

Post by adwn »

MainJane wrote: post #359
adwn wrote: post #358

Earlier quoted context omitted.

... and it uses a manually implemented post-processing step. You can't just run the sort program with GNU parallel and expect to get a fully sorted list.

> Try using GNU parallel to sort a list of numbers, [...] – it's not possible, Yet it clearly is possible, so your blanket statement is clearly wrong. `parsort` a simple wrapper, and this really goes for many uses of GNU Parallel: You need to prepare your data for the parallel step and post-process the output. Maybe you originally meant to say: "Only for those processes, where you can preprocess the input and post-pr…

Why would you use GNU parallel if you have to implement your own non-trivial pre- or post-processing logic anyway? Just spawn the worker processes yourself.

GNU parallel is great if you have, e.g., a bunch of files, each of which needs to be processed individually, like running awk or sed over it. Then you can just plop parallel in front and get a speedup for free. That's not what parsort does.

Post reply on HN