Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

241–250 of 366 posts

Re: A viable solution for Python concurrency

#241

Post by rtpg »

OOPMan wrote: post #234

It's weird how the more I work with Python, the less I want to work with Python. I moved into the language full-time in 2010 and it's now 2021. The packaging ecosystem is still a burning dumpster fire, the performance is still hot garbage and the whole approach to asyncio makes me want to bang my head against a wall. Tthe latest additions in Python 3.10 have me shaking my head. I love pattern matching (Yes, Scala fan…

Hyperfocusing on pattern matching and walrus operators when there are many extremely good improvements happening in Python fixing _real issues people have in production_ (stuff like packing in python timezones, improvements to pip's resolvers to make package resolution more correct, defaulting locale to UTF8) is very annoying.

Like the syntax is... divisive. And there's stuff like Black not being able to work with 3.10 for a while cuz it needs to rewrite its parser. But Python 2 -> 3 was a mess, and the python maintainers learned their lesson by making changes that would improve existing code without having to do full rewrites.

Like if you scroll through the changelog of recent releases you'll find loads of QoL improvements and things that are actually making the language more portable and easier to use in production. The CPython team does listen to needs of people (at least when it goes beyond aesthetics concerns...)

Re: A viable solution for Python concurrency

#242

Post by KZerda »

otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

Lots of reasons. Sometimes concurrency is a relatively soft need where you could get by with multiple processes, but it would be nice if the language itself provided some capability for things like threads. Or your dev team is much more familiar with python than with other languages, and the time to retrain and rewrite would be greater than the benefits. Or the rest of the language could have greater issues and you don't want to give up excellent libraries like numpy.

Re: A viable solution for Python concurrency

#243

Post by rtpg »

throwaway81523 wrote: post #164
BeetleB wrote: post #143

Earlier quoted context omitted.

You are likely being downvoted because most claims about the pain of a Python 3 transition are inflated/hyperbole. It took less than a day to migrate all my code to Python 3. And by "less than a day" I mean "less than 2 hours". Granted, bigger projects would take longer, but saying stuff like "10+ years of pain" is ridiculous. Probably less than 1% of projects had serious issues with the migration. We just hear of a…

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…

`.decode` doesn't work on strings because you decode bytes! PYthon has UTF strings!

I agree about the annoyance of the codec modules, and stuff like the urllib reorg happening on 3.0 instead of shifting it to later releases (and self-owns like the u prefix). But if you're calling `decode` on a string you have a bug in hiding, from the moment that your "code that handles encodings transparently" blows up when someone gives you a file that isn't UTF-encoded.

signed: someone who did the 2 -> 3 transition for a project that deals with a lot of non-UTF-stuff.

Re: A viable solution for Python concurrency

#244

Post by cormacrelf »

fatbird wrote: post #17
fatbird wrote: post #9

Earlier quoted context omitted.

It was Guido's requirement that GIL removal not degrade single threaded performance at all, but in the talk I attend at PyCon 2019, the speaker mentioned nothing about qualifications on that. Guido's restriction was presented, quite reasonably, as "no one should have to suffer because of removing the GIL". So a net break-even or performance improvement is fine. And on top of that, Guido has retired now, and the steer…

Guido has replied to Gross's announcement to observe that his performance improvements are not tied to removing the GIL and could be accepted separately. But he doesn't reject Gross's work outright, and if the same release that includes the GIL removal also delivers a concrete performance upgrade, I suspect that Guido would be fine with it. His concern is, after all, practical, to do with the actual use of python and…

I'm not saying this is a good idea (it is not, please don't do this to your open source community of choice!) but it'd be a pretty funny use of copyright licensing to reserve all rights on the non-GIL improvements unless the GIL ones were also merged. Strong-arming an open source project with sheer programming skill.

Re: A viable solution for Python concurrency

#245

Post by tehjoker »

game_the0ry wrote: post #200

I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is. Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that. Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And someti…

Python is used for high performance scientific computing... it's just all the HPC parts get written in C/C++ wrappers.

Re: A viable solution for Python concurrency

#246

Post by pansa2 »

OOPMan wrote: post #234

It's weird how the more I work with Python, the less I want to work with Python. I moved into the language full-time in 2010 and it's now 2021. The packaging ecosystem is still a burning dumpster fire, the performance is still hot garbage and the whole approach to asyncio makes me want to bang my head against a wall. Tthe latest additions in Python 3.10 have me shaking my head. I love pattern matching (Yes, Scala fan…

> The packaging ecosystem is still a burning dumpster fire, the performance is still hot garbage and the whole approach to asyncio makes me want to bang my head against a wall.

These issues, plus the inability to build a standalone binary, are why many Python programmers have adopted Go. Other than maybe packaging, Go gets all these things right. It’s just a shame that the language itself isn’t as nice as Python.

Re: A viable solution for Python concurrency

#247

Post by tialaramex »

zzzeek wrote: post #211
throwaway81523 wrote: post #164

Earlier quoted context omitted.

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…

And yet python tops the Tiobe index anyway. The python 3 transition was not that big a deal, and I migrated or assisted in the migration of many codebases some of which are as big as they get (openstack). the ten year thing especially made it really gradual, and we're all done now. Python 3 is great. Folks migrating to rust / go etc are looking at performance concerns for high volume server software, which python is…

However if you're confident Python simply is "never going to specialize in" performance, maybe the GIL isn't a problem and this work is irrelevant right?

I'm dubious about this work because I think it tries to paper over the cracks in code with data races, but maybe the problem was there already with the GIL and this changes little.

Re: A viable solution for Python concurrency

#248

Post by garmaine »

OOPMan wrote: post #236
rmdashrfstar wrote: post #235

Earlier quoted context omitted.

You’ll find camaraderie in the Rust community, we have similar stories… :) One of us, one of us, one of us!

Rust is on my list of things to look at one day, but I'm still on the Scala train for now ;-)

Another former Python dev, now rust dev… give it a try. I have trouble putting it in words, but Rust has the feeling of ease of expressiveness that makes Python fun to work with, but with a top notch static typing system. Lots of former Python devs doing rust work now.

Re: A viable solution for Python concurrency

#249

Post by garmaine »

rtpg wrote: post #243
throwaway81523 wrote: post #164

Earlier quoted context omitted.

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…

`.decode` doesn't work on strings because you decode bytes! PYthon has UTF strings! I agree about the annoyance of the codec modules, and stuff like the urllib reorg happening on 3.0 instead of shifting it to later releases (and self-owns like the u prefix). But if you're calling `decode` on a string you have a bug in hiding, from the moment that your "code that handles encodings transparently" blows up when someone…

So? `.decode(“hex”)` still makes sense on a string, and was >95% of the encode/decode calls I had.

Re: A viable solution for Python concurrency

#250

Post by slaymaker1907 »

throwaway81523 wrote: post #59
ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

Yes, I have a big sense of tragedy about Python 3. Python should run on something like (or maybe the actual) Erlang BEAM with lightweight isolated processes. All my threaded Python code is written using that style anyway (threads communicating through synchronized queues) and I've almost never needed traditional shared mutable objects. Maybe completely never, but I'm not sure about a certain program any more. Added:…

Yeah, I'd agree it's kind of stupid to use OS threads if you are going to have a GIL. It does make the implementation simpler, but it comes at tremendous cost to IO bound programs. If you are actually trying to do computationally intensive work, you should really be using multiple processes instead of threads in a language with GC. When writing a UI, moving work to a separate thread can still lag because GC will also block the UI thread. Even if you don't care about latency, having separate memory pools via separate processes often helps GC because then GC is embarrassingly parallel regardless of GC implementation.
Post reply on HN