Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

81–90 of 366 posts

Re: A viable solution for Python concurrency

#81

Post by kzrdude »

cogman10 wrote: post #47
a1369209993 wrote: post #34

> With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast, non-atomic instructions. Any other thread accessing the object will use atomic operations on the shared reference count. > Whenever the owning th…

Yeah, I'm not exactly getting all the complexity here. I'm digging the 2 reference counters, that makes sense to me, but I don't know why it isn't something more like: "every time a new thread takes a reference, atomic +1, every time a new thread's local count hits 0, atomic -1. If the shared reference is 0, free". IDK what special purpose the flags are serving here.

To have one local count per thread would add memory overhead, I think? In his solution there are only two counters per object, local and shared.

Any other thread can't know if it's the first time or not it's taking a reference to an object.

Re: A viable solution for Python concurrency

#82

Post by kzrdude »

Animats wrote: post #43
formerly_proven wrote: post #38

Earlier quoted context omitted.

You mean programs where you put an object into pickle and some other threads modify it while pickle is processing it? Doesn't surprise me - the equivalent written in plain Python would be very thread unsafe as well.

No, I mean several threads doing completely separate CPickle streams with no shared data or variables at the Python level.

Has it since been fixed?

Re: A viable solution for Python concurrency

#83

Post by BiteCode_dev »

dralley wrote: post #57
BiteCode_dev wrote: post #26

Earlier quoted context omitted.

asyncio is not a competition to threads, it's complementary. In fact, it's a perfectly viable strat in python to have several processes, each having several threads, each having an event loop. And it will still be so, once this comes out. You will certainly use threads more, and processes less, but replacing 1000000 coroutines by 1000000 system threads is not necessarily the right strategy for your task. See nginx vs…

Multiple threads with one asyncio loop per thread would be absolutely pointless in Python, because of the GIL. With that said, sure, threads and asyncio are complimentary in the sense that you can run tasks on threadpool executors and treat them as if they were coroutines on an event loop. But that serves no purpose unless you're trying to do blocking IO without blocking your whole process.

It would not be pointless at all, because while one thread may lock on CPU, context switching will let another one deal with IO. This can let you smooth out the progress of each part of your program, and can be useful for workload when you don't want anything to block for too long.

Re: A viable solution for Python concurrency

#84

Post by harpiaharpyja »

ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

If you are ever considering making use of asyncio for your project, I would strongly recommend taking a look at curio [1] as an alternative. It's like asyncio but far, far easier to use.

[1] https://curio.readthedocs.io/en/latest/index.html

Re: A viable solution for Python concurrency

#85

Post by BiteCode_dev »

Zarathust wrote: post #64
BiteCode_dev wrote: post #26

Earlier quoted context omitted.

asyncio is not a competition to threads, it's complementary. In fact, it's a perfectly viable strat in python to have several processes, each having several threads, each having an event loop. And it will still be so, once this comes out. You will certainly use threads more, and processes less, but replacing 1000000 coroutines by 1000000 system threads is not necessarily the right strategy for your task. See nginx vs…

"Viable" as in "you have no other choice sometimes". This forces you to deal with 3 libraries each with their own quirks, pitfalls and incompatibilities. Sometimes you even deal with dependencies reimplementing some parts in a 4th or 5th library to deal with shortcomings. I really don't care that much which of them survive, I just want to rely on less of them

No, it's just useful. They are techs with different trade off, and life is full of opportunities.

Re: A viable solution for Python concurrency

#87

Post by heavyset_go »

marris wrote: post #77

How big a problem is the possible breakage of C extensions for new code? Is there currently some standard "future proofed for multi-thread" way of writing them that will reduce the odds of the C extension breaking? And maybe also being compatible with PyPy? Or do developers today need to write a separate version for each interpreter that they want to support?

There are projects[1] that are abstracting away the C extension interface in order to standardize C extensions across implementations and prevent breaking changes.

[1] https://github.com/hpyproject/hpy

Re: A viable solution for Python concurrency

#88

Post by a1369209993 »

jeremyjh wrote: post #52
a1369209993 wrote: post #34

> With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast, non-atomic instructions. Any other thread accessing the object will use atomic operations on the shared reference count. > Whenever the owning th…

Threads don’t hold references - other objects do and we have to know how many do. If threads held a reference it might never be released. Since most objects are never shared we wouldn’t want to increment an atomic counter even once for those.

I don't think the objection you're actually making is valid (the extra atomic reference is just a representation of the fact that the local refcount is nonzero), but come to think of it, even in the original version, how the heck does a thread know whether a reference held by (say) a dictionary that is itself accesible to multiple threads was increfed by the owning thread or another thread?

Re: A viable solution for Python concurrency

#89

Post by a1369209993 »

dennisafa wrote: post #46
a1369209993 wrote: post #34

> With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast, non-atomic instructions. Any other thread accessing the object will use atomic operations on the shared reference count. > Whenever the owning th…

I think that idea was mentioned earlier in the article: > The simplest change would be to replace non-atomic reference count operations with their atomic equivalents. However, atomic instructions are more expensive than their non-atomic counterparts. Replacing Py_INCREF and Py_DECREF with atomic variants would result in a 60% average slowdown on the pyperformance benchmark suite.

> to replace non-atomic reference count operations with their atomic equivalents.

Nope, my proposal still uses two reference counts (one atomic, one local); it just avoids having a seperate flag bit to indicate that the owning thread is done.

Re: A viable solution for Python concurrency

#90

Post by heavyset_go »

overgard wrote: post #40

I feel like Gvr just doesnt want to change things, Feels doomed This has been a problem for like 20 years and they have refused fixes before. And there have been fixes. They just don't see this as important it's practically a religion that its a thing they wont change

I disagree entirely. The last few releases of Python have made significant changes to the language, coinciding with the project becoming community-led after Guido stepped down.
Post reply on HN