Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

111–120 of 366 posts

Re: A viable solution for Python concurrency

#111

Post by twic »

Jtsummers wrote: post #102
twic wrote: post #93

> With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast, non-atomic instructions. Any other thread accessing the object will use atomic operations on the shared reference count. > Whenever the owning th…

Two spaces in front of each line of the code block. As written, right now, your comment is hard to parse: import threading def produce(): global global_foo local_foo = "potato" global_foo = local_foo def consume(): global global_foo local_foo = global_foo global_foo = None if __name__ == '__main__': produce() thread = threading.Thread(target=consume) thread.start() thread.join()

Sorry about that. I did indent before pasting the code - but gedit indents with tabs, which HN ignores!

Re: A viable solution for Python concurrency

#112

Post by mixmastamyk »

heavyset_go wrote: post #90
overgard wrote: post #40

I feel like Gvr just doesnt want to change things, Feels doomed This has been a problem for like 20 years and they have refused fixes before. And there have been fixes. They just don't see this as important it's practically a religion that its a thing they wont change

I disagree entirely. The last few releases of Python have made significant changes to the language, coinciding with the project becoming community-led after Guido stepped down.

Yes, stepped down as a result of him forcing the walrus operator change into the language over significant opposition.

Re: A viable solution for Python concurrency

#113

Post by sandGorgon »

heavyset_go wrote: post #103
BiteCode_dev wrote: post #99

Earlier quoted context omitted.

While the design of Curio is quite interesting, it's may not be a good choice, not for technical reasons, but for logistical reasons: the chances it gets a wide adoption are slim to None. And since we are stuck with colored functions in python, the choice of stack matters very much. Now, if you want easier concurrency, and a solution to a lot of concurrency problems that curio solves, while still being compatible wit…

anyio is also compatible with asyncio and Trio, so you can use it with either library or paradigm.

Uvloop/uvicorn - which is the production grade asgi server only works with asyncio.

Hypercorn works with trio..but you lose a LOT of performance

Re: A viable solution for Python concurrency

#114

Post by ska »

otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

To a first approximation, people don't use python for itself, they use it for the vast ecosystem and network effect. If you jump to another language for better concurrency, what are you giving up?

Unless you really are doing greenfield development in an isolated application, these considerations often trump any language feature.

Re: A viable solution for Python concurrency

#115

Post by ferdowsi »

otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

There are some organizations with lots of domain knowledge and expertise around around developing, securing and deploying Python and they don't have the Innovation Currency to spend on investing in a new language.

Specific to your point, recruiting for Elixir talent is a problem compared to more mainstream languages. Recruiting in general is extremely hard at this moment.

Re: A viable solution for Python concurrency

#116

Post by pmontra »

otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

A possible answer is that everybody in the company knows Python and no other language. Another one is that they have to reuse or extend a bunch of existing Python code. The latter happened to me. Performances were definitely not a concern but I suddenly needed threads doing extra functionality over the original single threaded algorithm. BTW, I used a queue to pass messages between them.

Re: A viable solution for Python concurrency

#117

Post by otterley »

ferdowsi wrote: post #115
otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

There are some organizations with lots of domain knowledge and expertise around around developing, securing and deploying Python and they don't have the Innovation Currency to spend on investing in a new language. Specific to your point, recruiting for Elixir talent is a problem compared to more mainstream languages. Recruiting in general is extremely hard at this moment.

Given all the corner cases people are going to continue to find whilst trying to coax Python into behaving correctly in a highly concurrent program -- especially one that utilizes random libraries from the ecosystem -- I can't help but wonder whether the Innovation Currency is better spent replacing the components that require high concurrency (which often is only a subset of them) instead of getting stuck in the mire of bug-smashing.

Re: A viable solution for Python concurrency

#118

Post by Fordec »

otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

Sometimes a project starts off aiming to solve a problem. maybe it's a data science problem, so support already exists in python so lets do that. Ok, it worked great and it's catching on with users. Now we need to scale, but we are running into concurrency issues. What is a better answer? Ok we will work on improving python concurrency under the hood, or completely scrap the code base and switch to a different language?

Very few people set out going asking themselves about such low level details on day one of a project. Especially something that was an MVP or POC

Re: A viable solution for Python concurrency

#119

Post by otterley »

pmontra wrote: post #116
otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

A possible answer is that everybody in the company knows Python and no other language. Another one is that they have to reuse or extend a bunch of existing Python code. The latter happened to me. Performances were definitely not a concern but I suddenly needed threads doing extra functionality over the original single threaded algorithm. BTW, I used a queue to pass messages between them.

Using multiple interpreters with message passing is a workable, if expensive, way to deal with the problem. It is trading one cost for another. (These sort of tradeoffs are encountered all the time in business, to be sure.)

Re: A viable solution for Python concurrency

#120

Post by twic »

ameixaseca wrote: post #107
twic wrote: post #93

> With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast, non-atomic instructions. Any other thread accessing the object will use atomic operations on the shared reference count. > Whenever the owning th…

I couldn't find this in the design document but the only obvious solution is to track globals via the shared count. Since a global reference is part of all threads simultaneously, it cannot be treated as local. If you follow this reasoning, the operations above result in local=0/shared=0 after the last assignment.

As i said in the comment, that doesn't work. Put a list in the global, and then push and pop the string on the list. Even better, push the string into a local list, then put that list in another local list, then put that in a global, etc. You would need to dynamically keep every object reachable from a global marked as such, and that's a non-starter.
Post reply on HN