Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

201–210 of 366 posts

Re: A viable solution for Python concurrency

#201

Post by tbabb »

aeyes wrote: post #131
tbabb wrote: post #19

Earlier quoted context omitted.

What specifically is the problem with asyncio? I quite like using it, so I'm curious if there's some aspect that makes it unsustainable?

It solves only one problem, the name says it: Async I/O If you do anything on the CPU or if you have any I/O which is not async you stall the event loop and everything grinds to a halt. Imagine a program which needs to send heartbeats or data to a server in a short interval to show liveness, Kafka for example. Asyncio alone can't reliably do this, you need to take great care to not stall the event loop. You only have…

> If you do anything on the CPU or if you have any I/O which is not async you stall the event loop and everything grinds to a halt.

But that's always going to be the case for single threaded code. If you're ready to use threads, then that integrates pretty seamlessly with asyncio by using an executor task.

> asyncio doesn't give us any tools to protect the event loop from getting stalled by your code

Again, check out executors: https://docs.python.org/3/library/asyncio-eventloop.html#asy...

Asyncio in Python and JS are very different beasts.

Re: A viable solution for Python concurrency

#202

Post by heavyset_go »

dragonwriter wrote: post #195
heavyset_go wrote: post #90

Earlier quoted context omitted.

I disagree entirely. The last few releases of Python have made significant changes to the language, coinciding with the project becoming community-led after Guido stepped down.

> The last few releases of Python have made significant changes to the language, coinciding with the project becoming community-led after Guido stepped down. A lot of that is stuff that is enabled by, or was blocked pending, the new parser; I don't think it was blocked on Guido, and Guido haa hardly stopped being active and influential since stepping down as BDfL

Yeah, I didn't mean to imply that Guido was holding anything back, just that he's no longer at the helm like the OP implied.

Re: A viable solution for Python concurrency

#203

Post by ferdowsi »

game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

That's a fair opinion. Like many academics and data scientists are probably fine with Python as a tool for scripting or as a simple interface language for calling C libraries.

But if Python continues as is, it will continue to to lose major enterprise traction as web companies transition to using faster languages for applications and infrastructure. It'll be tough to stave off the negative feedback loop at that point.

Re: A viable solution for Python concurrency

#204

Post by hahajk »

game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

But what if we had both? What if Ruby was as fast as Java? What a wonderful world that would be!

Re: A viable solution for Python concurrency

#205

Post by unityByFreedom »

huetius wrote: post #186
unityByFreedom wrote: post #174

Earlier quoted context omitted.

This is more about making parallel python easier to use. It's already fast if you know about the current GIL workarounds. It'd be nice to not have to monkey patch, for example.

Quite true. Definitely a benefit. I guess my point is that if you are leaning on Python for performant concurrent operations, you are likely to also be thinking about how to isolate that component for a rewrite, monkey-patching or no.

> how to isolate that component

At the most basic level you only need to put the logic into a function. Of course, the cost of that adjustment may vary, so more options are good. If Python becomes easier to use in this respect then everyone wins.

Re: A viable solution for Python concurrency

#206

Post by arc-in-space »

game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

What do you mean? None of this is incompatible with being fast, and no changes to the language itself were being proposed here.

Re: A viable solution for Python concurrency

#207

Post by arc-in-space »

omnicognate wrote: post #183
lucb1e wrote: post #147

Earlier quoted context omitted.

For a minute I thought I finally found someone else who likes the GIL, but then you said content about . Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading…

I like the GIL and would prefer it not be removed, regardless of any impact on performance. It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's co…

I'm confused. The article mentions that Python's data structures would be made thread-safe, which sounds like there is no intention to change concurrency semantics. How does the presence of the GIL help? Can you elaborate what would be lost by removing it?

Re: A viable solution for Python concurrency

#208

Post by Twisol »

ferdowsi wrote: post #203
game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

That's a fair opinion. Like many academics and data scientists are probably fine with Python as a tool for scripting or as a simple interface language for calling C libraries. But if Python continues as is, it will continue to to lose major enterprise traction as web companies transition to using faster languages for applications and infrastructure. It'll be tough to stave off the negative feedback loop at that point…

I'm not confident that's a bad thing, and I worry that we're judging languages on metrics that only make sense for startups. Does a language need to chase growth?

I'm not saying Python shouldn't improve. But, if it comes to it, Python shouldn't cannibalize the niche it filled to do so. That will just spark other languages to fill the niche again.

Re: A viable solution for Python concurrency

#209

Post by wokwokwok »

game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

> I do not think python should try to be concurrent

Fair.

> or any more performant than it already is.

Well, you can certainly argue that you’d prefer that the effort devoted to performance could be better spent elsewhere, but it seems a little down to say you don’t think it should be faster.

Let’s not beat around the bush here: pure python is orders of magnitude slower than other similar tier languages, eg. javascript.

That’s because it’s never been a priority.

It also keeps getting new features.

You can’t keep adding new features to anything without it getting slower bloating and becoming sluggish unless you devote effort to keeping things fast.

So, I certainly have no problems with people working on making it faster.

That’s awesome.

The goal isn’t “pure python only, no more c!”

It’s just: stop the pure python code being such a major bottleneck.

Re: A viable solution for Python concurrency

#210

Post by arc-in-space »

solarmist wrote: post #158
masklinn wrote: post #30

Earlier quoted context omitted.

> My understanding was that CPython viewed any single-threaded performance regression as a blocker to GIL-removal attempts, regardless of if other work by the developer has sped up the interpreter? Previous GILectomy attempts incurred significant single-threaded performance penalties, on on the order of 50% or above. If Gross's work yields low single-digit performance penalty it's pretty likely to be accepted as this…

The article mentions a 10% improvement to single threaded runtime on average.

Yes, but this speedup comes from essentially unrelated changes from the same contributor, not from the parallelization work, which by itself would be a small slowdown (assuming the article is accurate).
Post reply on HN