Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Asyncio

superfastpython.com

Post reply on HN
Print view View original

131–140 of 188 posts

Re: Python Asyncio

#131

Post by m3047 »

gilad wrote: post #104
m3047 wrote: post #76

I seem to use asyncio a lot, so maybe it's just good for internet plumbing. Things I've used it for: * A Postfix TCP table. * A milter. * DNS request forwarding. * Reading data from a Unix domain socket and firing off dynamic DNS updates. * A DNS proxy for Redis. * A netflow agent. * A stream feeder for Redis. https://github.com/search?q=user%3Am3047+asyncio&type=Reposi... By the way you can't use it for disk I/O, bu…

what about aiofiles[0] for disk I/O? [0] https://github.com/Tinche/aiofiles

Not sure. A cursory look suggests it runs file ops in a thread pool.

The problem that I'm aware of is at a deeper level and has to do with the ability (or lack thereof) to set nonblocking on file descriptors associated with disk files.

Re: Python Asyncio

#132

Post by Steltek »

quietbritishjim wrote: post #10

Not complete - doesn't include Task Groups [1] In fairness they were only included in asyncio as of Python 3.11, which was released a couple of weeks ago. These were an idea originally from Trio [2] where they're called "nurseries" instead of "task groups". My view is that you're better off using Trio, or at least anyio [3] which gives a Trio-like interface to asyncio. One particularly nice thing about Trio (and anyi…

Python, more than other languages, ends up being used in utility scripts on end user devices, where you'll find just about everything. Python's async seems to still have a large swing in available features. Unlike async methods in other languages, semi-serious dabbling in it just for fun is courting a lot of headaches later.

Re: Python Asyncio

#133

Post by JosephRedfern »

number6 wrote: post #79
kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

Asyncio is not ment for CPU Bound Task but for IO Bound Tasks. You should have used multiprocessing. The problem you describe is exactly what asyncio is used for: saturate your bandwidth better. maybe the right aproach would have been a Threadpool? Plus you don't have to refractor the task. Just let it run sync, but you can make it also async

Depending on the specifics, a threadpool proabably wouldn't help with a CPU bound task due to GIL limitations. `multiprocess.Pool` might (though process pools aren' without their overheads)

Re: Python Asyncio

#134

Post by dekhn »

throwawaymaths wrote: post #21
acedTrex wrote: post #18

Earlier quoted context omitted.

This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.

> Async just makes that simpler to read and write. Really? I definitely had a huge struggle reading async python, because generators and yields break the concept of "what is a function" and struggled writing async python because of function coloring.

I am really not a fan of generators and yields for the same reason. But over time I've come to see it as ever-so-slightly more elegant syntax sugar for a class that maintains internal state to sequentially return specific values in response to a series of calls to a function.

Re: Python Asyncio

#135

Post by btown »

dekhn wrote: post #111

I've used python since 1995 and I can say that async is one of the worst things I've seen put into python since then. I've used a wide range of frameworks (twisted, gevent, etc) as well as threads and even if async is a good solution (I don't think it is) it broke awscli for quite some time (through aiobotocore and related package dependencies). It's too late in the game for long-term breaks like that or any backward…

Yep, it's 2022 and gevent is still the only solution for async & high concurrency that Just Works with the entire ecosystem of Python libraries without code changes. There's definitely some compute overhead compared to async, but we save so much developer time having effortless concurrency and never being worried that, say, using a slow third-party API over the web will slow down other requests.

Re: Python Asyncio

#137

Post by phs2501 »

dekhn wrote: post #127
number6 wrote: post #116

Earlier quoted context omitted.

I only know C#: https://learn.microsoft.com/en-us/dotnet/csharp/async#recogn... There you also have to act differently based the task

The docs on that page explicitly say you just have to provide a different parameter (basically, to force it to use a thread-like instead of select-like approach). The docs on that page also point you to a different task library which does what I'd expect: "If the work is appropriate for concurrency and parallelism, also consider using the Task Parallel Library." I checked those docs and that is a framework that makes…

Not sure how that's different than Python asyncio's "run blocking stuff with `run_in_executor`, the result (future) of which integrates into the async event loop?

Re: Python Asyncio

#138

Post by _bohm »

kissgyorgy wrote: post #80
nine_k wrote: post #59

Earlier quoted context omitted.

I think this may be influenced by Node. With JS and Node, async is cheap and ergonomic, so it's very widely used, even if the latency gains are marginal. With Python and asyncio, neither assumption holds, but the idea of shaving off some latency persists: if we rewrite everything using asyncio, our DB accesses can be parallelized, yay! This may or may not be a net win in latency; reworking your DB access patterns may…

> reworking your DB access patterns may gain you more. Nope: https://techspot.zzzeek.org/2015/02/15/asynchronous-python-a...

It may not be the norm, but there are definitely apps/services out there that require heavy-duty database queries and can have their throughput bottlenecked by synchronous DB IO latency, and would benefit from asyncio on the database side. Also, SQLAlchemy supports asyncio now.

Re: Python Asyncio

#139

Post by throwawaymaths »

dekhn wrote: post #134
throwawaymaths wrote: post #21

Earlier quoted context omitted.

> Async just makes that simpler to read and write. Really? I definitely had a huge struggle reading async python, because generators and yields break the concept of "what is a function" and struggled writing async python because of function coloring.

I am really not a fan of generators and yields for the same reason. But over time I've come to see it as ever-so-slightly more elegant syntax sugar for a class that maintains internal state to sequentially return specific values in response to a series of calls to a function.

The least they could have done is sugared it to something that isn't "def", like "gen" or "defgen"

Re: Python Asyncio

#140

Post by kissgyorgy »

mixmastamyk wrote: post #124
kissgyorgy wrote: post #26

Earlier quoted context omitted.

I'm pretty sure a lot of people don't understand the tradeoffs and don't really need concurrent code. There are really only a couple of use-cases where it's really handy: for example an API gateway, where you only get and send HTTP requests. Other than that, I don't see how it worth the insane complexity (compared to sync code). You can write web servers in a sync manner; Flask, Django is way better if you only need…

Choosing fastapi for a fast api is a mistake?

Choosing unnecessary complexity for a simple task is a mistake. FastAPI is not faster than Django or Flask for a single request, quite the opposite.
Post reply on HN