Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

301–310 of 366 posts

Re: A viable solution for Python concurrency

#301

Post by globular-toast »

sigotirandolas wrote: post #266
globular-toast wrote: post #181

Earlier quoted context omitted.

> Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be f…

I don't think the parent was proposing reinventing the wheel, Python has straightforward process parallelism support in the 'multiprocessing' library and for Python that's generally a better idea than GNU Parallel, IMO.

The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.

Re: A viable solution for Python concurrency

#302

Post by aidos »

omnicognate wrote: post #284
sherjilozair wrote: post #214

Earlier quoted context omitted.

You know it’s trivial to add a global lock to any concurrent program, right? What would you lose if other people started writing performant code?

This comment under the article gives an example: https://lwn.net/Articles/872961/ . Several other comments there also discuss this problem. Unless an extension explicitly releases the GIL it's not possible for the state of the interpreter to change during execution of its methods. That's an invariant that extensions rely on implicitly for safety in many ways and it's hard to imagine how one could make them safe witho…

Is there a workaround here where that locking can be applied automatically from above when an extension hasn’t explicitly said it’s GIL-free safe?

Re: A viable solution for Python concurrency

#303

Post by MaXtreeM »

Slight off topic but I am curious about using bits in an integer for flags. As the article mentions Gross uses 2 least significat bits for flags and the rest is an integer for reference counting. When someone considers whether to use most significant bits or least significant bits are there any major differences? Is is easier to implement or faster because of processor architectures/instruction sets to use least significnt bits or is that just a matter of choice?

Re: A viable solution for Python concurrency

#304

Post by otterley »

Mehdi2277 wrote: post #265
otterley wrote: post #260

Earlier quoted context omitted.

This doesn’t sound right. I think you’re mixing up parallelism/multi-core/linear resource scalability with concurrency. Usually you want high concurrency to handle multiplexing events where a thread or process per client would be waiting idle the majority of the time. ML is compute bound so by definition there wouldn’t be idle threads. And you can already get multicore work done in Python by simply using its venerabl…

I'm familiar with both. The person who's leading the GIL removal is one of the main authors of one of the two leading ML libraries in python (pytorch) and his motivation is for primarily performance for ML. He explicitly talks about multiprocessing and while it does often work well, there are several situations where it causes issues. I think best thing is to look at his arguments against it, https://docs.google.com/…

Thanks! I skimmed the document. It sounds like one of the biggest complications arises when multithreaded third-party C++ etc libraries call Python, rather than the other way around. That is a much more difficult problem to solve.

Re: A viable solution for Python concurrency

#305

Post by da39a3ee »

globular-toast wrote: post #181
lucb1e wrote: post #147

Earlier quoted context omitted.

For a minute I thought I finally found someone else who likes the GIL, but then you said content about . Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading…

> Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be f…

> GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel?

We’re not talking about writing scripts to run on your laptop. We’re talking about code written for production applications. Deploying GNU parallel to production nodes / containers would be a major change to production systems that may not be feasible and even if it is would come with a high cost in terms of added complexity, maintenance, and production troubleshooting.

Re: A viable solution for Python concurrency

#306

Post by da39a3ee »

lucb1e wrote: post #288
globular-toast wrote: post #181

Earlier quoted context omitted.

> Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be f…

That's actually what I'm doing a lot of the time. Or even just bash: for i in {1..threadcount}; do pypy my.py $i/$threadcount & done;

We’re not talking about how to write scripts to run on your laptop, we’re talking about production systems.

Re: A viable solution for Python concurrency

#307

Post by da39a3ee »

garmaine wrote: post #251
toyg wrote: post #132

Earlier quoted context omitted.

Probably not. CPickle is famously shunned by anyone who has to do serious, performance-critical serialization/deserialization.

What do people use instead?

JSON, protocol buffers, thrift, etc. Saving python-native objects such as functions, class instances etc is usually not the right thing to do in production code.

Re: A viable solution for Python concurrency

#308

Post by da39a3ee »

game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

Python is ubiquitous in machine learning and data science, and it is also heavily used for server-side web applications. The fact that you only use it for scripting isn’t relevant to how the language should evolve given its usage by humans in general.

Re: A viable solution for Python concurrency

#309

Post by omnicognate »

aidos wrote: post #302
omnicognate wrote: post #284

Earlier quoted context omitted.

This comment under the article gives an example: https://lwn.net/Articles/872961/ . Several other comments there also discuss this problem. Unless an extension explicitly releases the GIL it's not possible for the state of the interpreter to change during execution of its methods. That's an invariant that extensions rely on implicitly for safety in many ways and it's hard to imagine how one could make them safe witho…

Is there a workaround here where that locking can be applied automatically from above when an extension hasn’t explicitly said it’s GIL-free safe?

Yes, but the locking inherently has to be global so it would have to be effectively the actual GIL if needed. The interpreter would therefore have to have a gil/nogil mode, maybe switched by a command-line parameter (you wouldn't want gil mode to be implicitly enabled interpreter-wide just because you imported a particular module). That's certainly possible, but I doubt it would be popular.

Re: A viable solution for Python concurrency

#310

Post by adwn »

globular-toast wrote: post #301
sigotirandolas wrote: post #266

Earlier quoted context omitted.

I don't think the parent was proposing reinventing the wheel, Python has straightforward process parallelism support in the 'multiprocessing' library and for Python that's generally a better idea than GNU Parallel, IMO.

The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.

> that works for any non-parallel process

No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.

Post reply on HN