Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Asyncio

superfastpython.com

Post reply on HN
Print view View original

101–110 of 188 posts

Re: Python Asyncio

#101

Post by walls »

kissgyorgy wrote: post #26
acedTrex wrote: post #18

Earlier quoted context omitted.

This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.

I'm pretty sure a lot of people don't understand the tradeoffs and don't really need concurrent code. There are really only a couple of use-cases where it's really handy: for example an API gateway, where you only get and send HTTP requests. Other than that, I don't see how it worth the insane complexity (compared to sync code). You can write web servers in a sync manner; Flask, Django is way better if you only need…

What kind of APIs are you writing that don’t involve db lookups or calls to other services?

Re: Python Asyncio

#102

Post by doliveira »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

The clue is in the name: asyncIO is for, well, I/O bound stuff...

Re: Python Asyncio

#103

Post by matsemann »

number6 wrote: post #79
kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

Asyncio is not ment for CPU Bound Task but for IO Bound Tasks. You should have used multiprocessing. The problem you describe is exactly what asyncio is used for: saturate your bandwidth better. maybe the right aproach would have been a Threadpool? Plus you don't have to refractor the task. Just let it run sync, but you can make it also async

In other languages the async paradigm work for multiple kind of workflows, not just heavily IO bound ones.

Re: Python Asyncio

#104

Post by gilad »

m3047 wrote: post #76

I seem to use asyncio a lot, so maybe it's just good for internet plumbing. Things I've used it for: * A Postfix TCP table. * A milter. * DNS request forwarding. * Reading data from a Unix domain socket and firing off dynamic DNS updates. * A DNS proxy for Redis. * A netflow agent. * A stream feeder for Redis. https://github.com/search?q=user%3Am3047+asyncio&type=Reposi... By the way you can't use it for disk I/O, bu…

what about aiofiles[0] for disk I/O?

[0] https://github.com/Tinche/aiofiles

Re: Python Asyncio

#105

Post by powersnail »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

Async in general (not just python) doesn’t speed up any computation.

The value is allowing you to do something else while waiting for IO to finish. If you are constantly doing IO, async is a good tool. Otherwise, it won’t help.

Re: Python Asyncio

#106

Post by [deleted] »

kissgyorgy wrote: post #80
nine_k wrote: post #59

Earlier quoted context omitted.

I think this may be influenced by Node. With JS and Node, async is cheap and ergonomic, so it's very widely used, even if the latency gains are marginal. With Python and asyncio, neither assumption holds, but the idea of shaving off some latency persists: if we rewrite everything using asyncio, our DB accesses can be parallelized, yay! This may or may not be a net win in latency; reworking your DB access patterns may…

> reworking your DB access patterns may gain you more. Nope: https://techspot.zzzeek.org/2015/02/15/asynchronous-python-a...

[deleted]

Re: Python Asyncio

#107

Post by nine_k »

kissgyorgy wrote: post #80
nine_k wrote: post #59

Earlier quoted context omitted.

I think this may be influenced by Node. With JS and Node, async is cheap and ergonomic, so it's very widely used, even if the latency gains are marginal. With Python and asyncio, neither assumption holds, but the idea of shaving off some latency persists: if we rewrite everything using asyncio, our DB accesses can be parallelized, yay! This may or may not be a net win in latency; reworking your DB access patterns may…

> reworking your DB access patterns may gain you more. Nope: https://techspot.zzzeek.org/2015/02/15/asynchronous-python-a...

I don't say that asyncio gives you nothing! Go for it with your DB access if you have already gone async in your design.

But often the lower-hanging fruit is doing more joins in the DB, fetching fewer columns, and writing your query more thoughtfully. Async only helps if you can do something while waiting for the DB to complete your query. A common anti-pattern, exacerbated by ORMs, is doing a bunch of small queries while passing bits of data between them in Python; moving that to asyncio won't help much, if any.

Re: Python Asyncio

#108

Post by rpep »

matsemann wrote: post #99
xwowsersx wrote: post #86

I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into it and tried to get some (what I think are) very basic examples working and really struggled. That alone is not fully dispositive because I've used many async implementations in various languages and each of them have a bit of a learning curve and their own wri…

Asyncio is annoying, and often unexpectedly slow. You think things are parallel, but then one misbehaving coroutine can hog your cpu bringing everything to a halt. GIL makes it useless for anything other than _heavily_ IO bound tasks. And yeah, Python's documentation is useless. Never how to use stuff, only listing of everything that's possible to do / the API. Unfortunately that style is being mimicked by most other…

I think the big point of interest will come with Python 3.12 where it's proposed that as part of the faster CPython project, the GIL-per-process will be removed.

Re: Python Asyncio

#109

Post by number6 »

matsemann wrote: post #103
number6 wrote: post #79

Earlier quoted context omitted.

Asyncio is not ment for CPU Bound Task but for IO Bound Tasks. You should have used multiprocessing. The problem you describe is exactly what asyncio is used for: saturate your bandwidth better. maybe the right aproach would have been a Threadpool? Plus you don't have to refractor the task. Just let it run sync, but you can make it also async

In other languages the async paradigm work for multiple kind of workflows, not just heavily IO bound ones.

Can you give me an example which languages do this?

Re: Python Asyncio

#110

Post by nine_k »

brrrrrm wrote: post #77
nine_k wrote: post #53

Earlier quoted context omitted.

How would ML benefit from async? I thought everything interesting in things like TF or PyTorch is done by native code which is free to use multithreading and anything at all. (I know little about ML.)

Specifically distributed would be helped (multiple nodes). Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (everything is blocking). If your hardware is non-homogenous (both interconnects and accelerators) or your program needs to be run in a pipelined way, you're fighting an extremely uphill battle. Go, JavaScript etc are the usual languages of choice for this type…

I suspect multiprocessing + shared memory could help this. The stdlib has provisions for both, but a coordination layer is needed.

That would be closest to true multithreading.

Post reply on HN