Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

291–300 of 366 posts

Re: A viable solution for Python concurrency

#291

Post by BiteCode_dev »

quietbritishjim wrote: post #289
BiteCode_dev wrote: post #101

Earlier quoted context omitted.

GOTO cosplaying should go away with structured concurrency (via TaskGroup) being adopted in 3.11, as pioneered by Trio. Check out anyio if you want to use them now.

TaskGroup in asyncio has been promised at least as far back as Python 3.8 [1]. They're still not in the draft "What's new in python 3.11" [2] and searching the web didn't return any official statements. I believe they're planned but don't believe they'll arrive any time soon. If you want to use structured concurrency now, IMO the best bet is to use Trio directly. Reading posts by the author makes it clear that every…

Anyio is probably a best bet to use TaskGroup IMO. You can use the trio backend if you want, after all.

But it's asyncio compatible, which makes it more future proof.

Re: A viable solution for Python concurrency

#292

Post by BiteCode_dev »

justusw wrote: post #233
BiteCode_dev wrote: post #76

Earlier quoted context omitted.

I used them both extensively, and here are the main reasons I can think of: - The event loop in JS is invisible and implicit. V8 proved it can be done without paying a cost for it, and in fact most real life python projects are using uvloop because it's faster than asyncio default loop. JS dev don't think of the loop at all, because it's always been there. They don't have to chose a loop, or thinking about its lifecy…

I’m not an expert on Node.js but I was always under the impression that you couldn’t write await outside of an async function. Has that changed recently?

You can await at the top level of a module since node v14.8.

It has been proposed to Python: https://groups.google.com/g/python-ideas/c/PN1_j7Md4j0/m/0xy...

But I have no more info about it.

Re: A viable solution for Python concurrency

#293

Post by BiteCode_dev »

throwaway81523 wrote: post #148
BiteCode_dev wrote: post #139

Earlier quoted context omitted.

It's a technical thread, not a political one. If you were so sure of your argument, you wouldn't use a throwaway. Besides, it's weird, like saying we should not have int, float and complex, there should be one way to do it. Just because those are 3 numbers doesn't mean they don't have each their own specific benefit.

int, float, and complex are for different purposes. async and threads paper over each others' weaknesses, instead of fixing the weaknesses at the start. Async itself is an antipattern (technical opinion, so there) but Python uses it because of the hazards and high costs of threads. Chuck Moore figured out 50 years ago to keep the async stuff out of the programmer's way, when he put multitasking into Polyforth, which…

Oh, you meant like, why don't Python didn't reimplement the whole interpreter around concurrency instead of using the tools it already had to find solutions to problems?

Well, that question is as old as engineering itself, and it's always a matter of resources, cost, history and knowledge.

Re: A viable solution for Python concurrency

#295

Post by andrewstuart »

ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

>>> In an alternate Python timeline, asyncio was not introduced into the Python standard library,

I totally disagree with this negative perspective of asyncio.

async programming is MUCH easier to program and understand than other concurrency solutions like multithreading.

async works extremely well as a programming model as evidenced by the fact that it's exactly the model being adopted by all languages that want to implement practical and developer friendly concurrency.

When I read such a negative take on asyncio I assume only this is a developer who hasn't done alot of programming with it, and is therefore still somewhat lacking in full understanding.

Re: A viable solution for Python concurrency

#296

Post by masklinn »

BiteCode_dev wrote: post #76
nine_k wrote: post #41

Earlier quoted context omitted.

BTW I wonder why async is so painless in ES6 compared to Python. Why the presence of GIL (which JS also has) did not make running async coroutines completely transparent, as it made running generators (which are, well, coroutines already). Why the whole even loop thing is even visible at all.

I used them both extensively, and here are the main reasons I can think of: - The event loop in JS is invisible and implicit. V8 proved it can be done without paying a cost for it, and in fact most real life python projects are using uvloop because it's faster than asyncio default loop. JS dev don't think of the loop at all, because it's always been there. They don't have to chose a loop, or thinking about its lifecy…

> Python is my favorite language, and I can live with the explicit loop, but explicit scheduling is ridiculous. Just run the damn coroutine

You can't "just run the damn coroutine", that's not what coroutines are.

But that does point to the mistake of Python, at least from a UX perspective: coroutines are more efficient but they're also a lot more prone to use errors, especially in dynamically typed languages. Async functions should probably just have been tasks, which is what they are in JS.

Re: A viable solution for Python concurrency

#297

Post by rylact »

nmca wrote: post #294

I work a non-FB $BIGCORP with lots of python super-experts who are very excited about this. Sam is deeply familiar with pytorch and C extensions; compatibility here seems to be the real deal. High hopes!

I'm completely new to pytorch (I looked at it for several hours today). To put it in a neutral manner, it seems to have a rather high tolerance for complexity and relying on dozens of external packages, including pybind11. Memory leaks included, as a cursory Google search reveals.

I hope this new style of writing Python packages does not leak into the interpreter.

Re: A viable solution for Python concurrency

#298

Post by sandGorgon »

I think this is a GREAT time to be doing this. This will undoubtedly shake the c-extension ecosystem. But it is already going to shake up because of Hpy (https://lwn.net/Articles/851202/)

So might as well do it in one shot.

But what im really interested in is - if this can be ported to Pypy. Given that pypy is already quite a bit faster than cpython...it would be interesting to see what the nogil will unleash

Re: A viable solution for Python concurrency

#299

Post by masklinn »

cletus wrote: post #226

This feels like Schrodinger's Cake to me (you know, having it and eating it too). > ... the first of which is called "biased reference counts" ... With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast,…

> So if the owner needs to check the shared ref count every time it changes the count of the non-atomic local count

It does not, it only needs to check the shared refcount when the local count reaches 0, to decide whether it can free the object immediately or whether it should mark it as "unowned".

Re: A viable solution for Python concurrency

#300

Post by garmaine »

rtpg wrote: post #264
garmaine wrote: post #249

Earlier quoted context omitted.

So? `.decode(“hex”)` still makes sense on a string, and was >95% of the encode/decode calls I had.

Hex-encoding works off of bytes… like if a character has a different byte representation between two encodings you need to specify in both directions Even the Python 2 strategy for hex might lead to your text getting garbled if you’re using anything not représentable in ASCII (roughly).

That’s not what decode(“hex”) does though. It takes strings in whatever encoding and turns 0-9,a-f into bytes. In Python 3 it would presumably return a bytes object instead, which you could then hypothetically call encode(“hex”) on. But the point is that it would have kept the syntax the same even if the underlying types changed.

Python3 unnecessarily broke compatibility in many places. Yes I agree that using bytes.fromhex() and .hex() are better design choices. But requiring manual inspection and updating of all strong manipulation code for the transition was an outrageous requirement, particularly for a dynamic language that doesn’t (or at least didn’t at that time) have compile-time type checking to catch such errors.

Post reply on HN