Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

331–340 of 366 posts

Re: A viable solution for Python concurrency

#331

Post by ryanpetrich »

MaXtreeM wrote: post #303

Slight off topic but I am curious about using bits in an integer for flags. As the article mentions Gross uses 2 least significat bits for flags and the rest is an integer for reference counting. When someone considers whether to use most significant bits or least significant bits are there any major differences? Is is easier to implement or faster because of processor architectures/instruction sets to use least sign…

A benefit of storing the reference count in the high bits is that overflows will never corrupt the flag bits and can be detected using the processor's flags instead of requiring a separate check. I'm not sure if this property is used here.

Re: A viable solution for Python concurrency

#332

Post by BeFlatXIII »

rtpg wrote: post #241
OOPMan wrote: post #234

It's weird how the more I work with Python, the less I want to work with Python. I moved into the language full-time in 2010 and it's now 2021. The packaging ecosystem is still a burning dumpster fire, the performance is still hot garbage and the whole approach to asyncio makes me want to bang my head against a wall. Tthe latest additions in Python 3.10 have me shaking my head. I love pattern matching (Yes, Scala fan…

Hyperfocusing on pattern matching and walrus operators when there are many extremely good improvements happening in Python fixing _real issues people have in production_ (stuff like packing in python timezones, improvements to pip's resolvers to make package resolution more correct, defaulting locale to UTF8) is very annoying. Like the syntax is... divisive. And there's stuff like Black not being able to work with 3.…

Could you tell us more about the "packing in Python timezones"? I'm curious to learn more (and it may be of tangential interest to one of my projects).

Re: A viable solution for Python concurrency

#333

Post by mixmastamyk »

the__alchemist wrote: post #255

Maybe we just accept that Python isn't suitable for concurrency. There's a community of Python developers who don't want to branch out; write everything in Python and never learn or consider another language. Let them be. Let Python excel at its core competencies; use the right tool for the job.

Odd time to write this comment as the first viable solution drops.

Re: A viable solution for Python concurrency

#334

Post by quietbritishjim »

BiteCode_dev wrote: post #291
quietbritishjim wrote: post #289

Earlier quoted context omitted.

TaskGroup in asyncio has been promised at least as far back as Python 3.8 [1]. They're still not in the draft "What's new in python 3.11" [2] and searching the web didn't return any official statements. I believe they're planned but don't believe they'll arrive any time soon. If you want to use structured concurrency now, IMO the best bet is to use Trio directly. Reading posts by the author makes it clear that every…

Anyio is probably a best bet to use TaskGroup IMO. You can use the trio backend if you want, after all. But it's asyncio compatible, which makes it more future proof.

Hmm, we'll just have to disagree on that one.

Trio has the better API, so you might as well use that directly rather some other library that attempts to match it, even if it can/does use it under the hood. It reminds me of that xkcd about standards. It's also going to have to settle for the least common denominator to some extent even if it reimplements some parts itself (e.g. if one library implements serial ports but the other didn't, then it surely can't be in anyio). I also dispute that it's "more future proof", because you're still relying on anyio being maintained, and to me it seems to be the more obscure library.

Re: A viable solution for Python concurrency

#335

Post by OOPMan »

BeFlatXIII wrote: post #332
rtpg wrote: post #241

Earlier quoted context omitted.

Hyperfocusing on pattern matching and walrus operators when there are many extremely good improvements happening in Python fixing _real issues people have in production_ (stuff like packing in python timezones, improvements to pip's resolvers to make package resolution more correct, defaulting locale to UTF8) is very annoying. Like the syntax is... divisive. And there's stuff like Black not being able to work with 3.…

Could you tell us more about the "packing in Python timezones"? I'm curious to learn more (and it may be of tangential interest to one of my projects).

They just bundled the timezone database in with the language. It's not particularly groundbreaking...

Re: A viable solution for Python concurrency

#336

Post by OOPMan »

garmaine wrote: post #248
OOPMan wrote: post #236

Earlier quoted context omitted.

Rust is on my list of things to look at one day, but I'm still on the Scala train for now ;-)

Another former Python dev, now rust dev… give it a try. I have trouble putting it in words, but Rust has the feeling of ease of expressiveness that makes Python fun to work with, but with a top notch static typing system. Lots of former Python devs doing rust work now.

Sounds like similar reasons why I enjoy Scala.

It feels like Python, but with a proper type system :-)

Re: A viable solution for Python concurrency

#337

Post by aidos »

omnicognate wrote: post #309
aidos wrote: post #302

Earlier quoted context omitted.

Is there a workaround here where that locking can be applied automatically from above when an extension hasn’t explicitly said it’s GIL-free safe?

Yes, but the locking inherently has to be global so it would have to be effectively the actual GIL if needed. The interpreter would therefore have to have a gil/nogil mode, maybe switched by a command-line parameter (you wouldn't want gil mode to be implicitly enabled interpreter-wide just because you imported a particular module). That's certainly possible, but I doubt it would be popular.

I guess that’s what I mean, yeah. For people that just don’t need these extensions, you could run without the GIL and then if it was required you could start using it while interacting with unsupported extensions.

I get what you’re saying, but for a lot of us, just being able to do nice multithreading for io would be a great enhancement. We have a use case when we’d like a bunch of threads to search through a large numpy structure and at the moment we have to stop to multiprocessing, which works. But is really heavy.

Re: A viable solution for Python concurrency

#338

Post by modeless »

azinman2 wrote: post #330
modeless wrote: post #262

Threads are certainly important, but I have to say that I found the multiprocessing package to work very well. I think a lot of the things people think they need threads for would actually be better with multiprocessing instead. Memory protection is good! Shared memory is still available and explicit sharing of just what you need is better in a lot of ways than implicit sharing of everything. I will be glad if the GI…

If the GIL is fixed, why would you want multiprocessing versus threads? Threads are cheaper to create, easier to communicate between (even if you need to be careful), and simply do different things than what multiprocessing intends (eg easier for blocking I/O on many threads, versus multiprocessing which is really more of a task queue)

Why don't we just run all code in different threads of the same process? Multiple processes are more robust to failure and easier to reason about because they are less tightly coupled and more explicit about sharing. You can do blocking I/O with multiprocessing, you just have to explicitly share buffers.

Re: A viable solution for Python concurrency

#339

Post by a1369209993 »

kccqzy wrote: post #223
jeremyjh wrote: post #52

Earlier quoted context omitted.

Threads don’t hold references - other objects do and we have to know how many do. If threads held a reference it might never be released. Since most objects are never shared we wouldn’t want to increment an atomic counter even once for those.

I think you have misunderstood GP. GP is trying to say, the thread that created the object has a remote (atomic) reference count of 1, in addition to a local reference count of 1. the remote reference count is simply initialized to be 1. During this initialization operation no atomic operation is used. The only atomic operation is during destruction: the local reference count is decremented non-atomically, and then i…

> and then if it is zero, we need an atomic memory_order_release decrement for the remote reference count.

Actually, we don't. We need a atomic load, but if the results of that load indicate that we're destroying the last reference to the object (which does need to be more complicated than just a extra remote reference; see [0] and [1]), we can just destroy the object as-is without actually doing a atomic store, because by definition there is no reference to the object through which another thread could inspect the remote refcount. (If there was, our reference wouldn't be the last one.)

0: https://news.ycombinator.com/item?id=28883218

1: https://sci-hub.se/https://dl.acm.org/doi/10.1145/3243176.32...

Re: A viable solution for Python concurrency

#340

Post by Matthias247 »

aeyes wrote: post #131
tbabb wrote: post #19

Earlier quoted context omitted.

What specifically is the problem with asyncio? I quite like using it, so I'm curious if there's some aspect that makes it unsustainable?

It solves only one problem, the name says it: Async I/O If you do anything on the CPU or if you have any I/O which is not async you stall the event loop and everything grinds to a halt. Imagine a program which needs to send heartbeats or data to a server in a short interval to show liveness, Kafka for example. Asyncio alone can't reliably do this, you need to take great care to not stall the event loop. You only have…

That's a problem - but its one that is common to all kinds of framework which make use of cooperative multitasking - whether they are in Python, Java (Netty, NIO, etc), C (Nginx, libevent, ...), C# or Rust.

All those frameworks trade off high performance if everything is well behaved against easy of use and potential of very bad performance if some regression is introduced.

Whether that tradeoff ia a good one to take will depend on the application.

Post reply on HN