A viable solution for Python concurrency • Hacker News Board

Re: A viable solution for Python concurrency

#211

Post by zzzeek » Sat, Oct 16, 2021, 12:51 AM UTC

throwaway81523 wrote: post #164

BeetleB wrote: post #143

Earlier quoted context omitted.

You are likely being downvoted because most claims about the pain of a Python 3 transition are inflated/hyperbole. It took less than a day to migrate all my code to Python 3. And by "less than a day" I mean "less than 2 hours". Granted, bigger projects would take longer, but saying stuff like "10+ years of pain" is ridiculous. Probably less than 1% of projects had serious issues with the migration. We just hear of a…

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…

And yet python tops the Tiobe index anyway. The python 3 transition was not that big a deal, and I migrated or assisted in the migration of many codebases some of which are as big as they get (openstack). the ten year thing especially made it really gradual, and we're all done now. Python 3 is great. Folks migrating to rust / go etc are looking at performance concerns for high volume server software, which python is simply never going to specialize in, and for which rust and go were both introduced only very recently. They didn't switch because python 3 was so disappointing.

Re: A viable solution for Python concurrency

#212

Post by js2 » Sat, Oct 16, 2021, 1:03 AM UTC

Buttons840 wrote: post #175

btown wrote: post #97

Earlier quoted context omitted.

> instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work Minus the "natively supported" part, we have this today in http://www.gevent.org/ ! It's so, so empowering to be able to access the entire historical body of work of synchronous-I/O Python libraries, and with a single monkey patch cause every I/O operation,…

> our web servers and batch workers have throughput limited only by CPU and RAM Are you able to fully utilize a multicore processor? I'm not familiar with gevent.

This is why you run one process per core, and you'll typically have something like nginx+uWSGI distribute requests across them. I use this combination with https://falconframework.org/ and boto3 to spool HTTP POST requests to S3 and SQS and am pretty happy with it.

uWSGI supports gevent https://uwsgi-docs.readthedocs.io/en/latest/Gevent.html

Falcon also benefits from Cython acceleration. It's been a while but at the time I tested against PyPy and either it was slower or had quirks I was unable to resolve (don't recall w/o consulting my notes).

Re: A viable solution for Python concurrency

#213

Post by beebmam » Sat, Oct 16, 2021, 1:06 AM UTC

throwaway81523 wrote: post #59

ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

Yes, I have a big sense of tragedy about Python 3. Python should run on something like (or maybe the actual) Erlang BEAM with lightweight isolated processes. All my threaded Python code is written using that style anyway (threads communicating through synchronized queues) and I've almost never needed traditional shared mutable objects. Maybe completely never, but I'm not sure about a certain program any more. Added:…

I think some people are upset about the irony/sarcasm here, and many people don't appreciate ironic/sarcastic posts (but I do!)

Re: A viable solution for Python concurrency

#214

Post by sherjilozair » Sat, Oct 16, 2021, 1:08 AM UTC

omnicognate wrote: post #183

lucb1e wrote: post #147

Earlier quoted context omitted.

For a minute I thought I finally found someone else who likes the GIL, but then you said content about . Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading…

I like the GIL and would prefer it not be removed, regardless of any impact on performance. It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's co…

You know it’s trivial to add a global lock to any concurrent program, right? What would you lose if other people started writing performant code?

Re: A viable solution for Python concurrency

#215

Post by dehrmann » Sat, Oct 16, 2021, 1:09 AM UTC

Animats wrote: post #4

Or you could just use PyPy, which uses a garbage collector, does more compile-time analysis, and runs much faster. CPython is a naive interpreter, like original JavaScript. There's been progress since then.

There's also Jython. If it weren't for Python's love of Cython and Jython being stuck at 2.7, it would be very viable.

Re: A viable solution for Python concurrency

#216

Post by sherjilozair » Sat, Oct 16, 2021, 1:09 AM UTC

zinodaur wrote: post #145

efoto wrote: post #3

"The biggest source of problems might be multi-threaded programs with concurrency-related bugs that have been masked by the GIL until now."

> concurrency-related bugs that have been masked by the GIL Yeah... could phrase this as "All programs written with the assumption of a GIL are now broken" instead. Wish they had done this as part of the breaking changes for python 3, I guess they'll have to wait for Python 4 for this?

GIL can remain default on. Users can simply disable it for chosen parts of their program.

Re: A viable solution for Python concurrency

#217

Post by sgerenser » Sat, Oct 16, 2021, 1:23 AM UTC

loeg wrote: post #189

rzimmerman wrote: post #177

Earlier quoted context omitted.

Objective C does (explicit) reference counting, though I suppose that's a pretty limited use case these days.

Doesn't Swift (built on ObjC) also use reference counting?

Yes, Swift uses refcounting.

Re: A viable solution for Python concurrency

#218

Post by pansa2 » Sat, Oct 16, 2021, 1:29 AM UTC

sgerenser wrote: post #217

loeg wrote: post #189

Earlier quoted context omitted.

Doesn't Swift (built on ObjC) also use reference counting?

Yes, Swift uses refcounting.

Does Swift have a GIL? If not, how does it solve the problem of multithreaded reference counting?

Re: A viable solution for Python concurrency

#219

Post by rattray » Sat, Oct 16, 2021, 1:32 AM UTC

game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

> Python is the tool I pick ... not for highly performant systems

Many high-throughput, high-concurrency systems run on Python, famously including Instagram and Pinterest (which see quite a lot of traffic).

Re: A viable solution for Python concurrency

#220

Post by Mehdi2277 » Sat, Oct 16, 2021, 1:32 AM UTC

otterley wrote: post #127

ska wrote: post #114

Earlier quoted context omitted.

To a first approximation, people don't use python for itself, they use it for the vast ecosystem and network effect. If you jump to another language for better concurrency, what are you giving up? Unless you really are doing greenfield development in an isolated application, these considerations often trump any language feature.

Don't get me wrong; I'm not suggesting that anyone dump Python altogether to switch to a different language for any arbitrary project or purpose. Many businesses I work with use different languages for different components or applications, using the network or storage (or even shared memory) to intercommunicate when necessary. The right tool for the job, as it were.

I use python mostly for numpy/tensorflow/etc. The machine learning ecosystem. That ecosystem cares a lot about multithreaded performance. So historically answer has been write c/c++ and then bind to python. This work is mainly motivated by libraries like that wanting to write less c extensions and be able to just write python and still have proper thread performance.