171–180 of 188 posts
Earlier quoted context omitted.
what about aiofiles[0] for disk I/O? [0] https://github.com/Tinche/aiofiles
Not sure. A cursory look suggests it runs file ops in a thread pool. The problem that I'm aware of is at a deeper level and has to do with the ability (or lack thereof) to set nonblocking on file descriptors associated with disk files.
If we had those, all this async idiocy would go away, no more code colours, no more single-core, even better isolation and protection against context-switching tangles.
(the implementation not so nice though. Python threads on machine threads ? Whose stupid idea was that ?)
Just in time, I've been needing to implement s3 upload asynchronously.
Earlier quoted context omitted.
Choosing fastapi for a fast api is a mistake?
Choosing unnecessary complexity for a simple task is a mistake. FastAPI is not faster than Django or Flask for a single request, quite the opposite.
Earlier quoted context omitted.
That's a blast from the past. Incredible that 10 years ago people thought using a thread per request is a good idea because anything else is too hard.
I'd argue that thread-per-request is even a better idea now, given the massive number of cores and memory in modern servers.
Request passed to a thread pool. Each thread is juggling coroutines representing in flight requests
Earlier quoted context omitted.
What kind of APIs are you writing that don’t involve db lookups or calls to other services?
Asyncio does not make your db calls or io faster, it does make your code more complex. The only thing async io gives you is scalability of many MANY concurrent io operations. You have to either be pushing seriously large traffic or doing a lot of long running websocket/sse/long polling type requests. The only other use case is lots of truly concurrent io within one request/response cycle. But again that is unusual, m…
Why just "within one cycle"? What about situation where you spend 1s in single db query and are 100ms cpu bound during request handling?
Earlier quoted context omitted.
I have such a feeling of tragedy about Python. I wish it had migrated to BEAM or implemented something similar, instead of growing all this async stuff. Whenever I see anything about Python asyncio, I'm reminded of gar1t's hilarious but NSFW rant about node.js, https://www.youtube.com/watch?v=bzkRVzciAZg . Content warning: lots of swearing, mostly near the end.
That's a blast from the past. Incredible that 10 years ago people thought using a thread per request is a good idea because anything else is too hard.
Earlier quoted context omitted.
I think the issue is more like this: 1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio) 2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword) 3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner 4. You end up having several mee…
Depending on the details you can often call an async function from a sync function, by just running the event loop, it is just a few lines of code. The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using.…
This works well if the callback is just there to notify you about a result (which is the most common case in fairness) but doesn't really work if the callback is meant to synchronously determine and return a result that is then used within the function that's calling the callback.
After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…
After using gevent for about a decade I have started using asyncio for new projects, just because it's in the standard library and has the official async blessing of the Python gods. Indeed it is way harder. I'm always coming up against little gotchas that take time to debug and fix. Part of me enjoys the challenge of learning something new and solving little puzzles. It's getting easier, especially as I build up a c…
I've used python since 1995 and I can say that async is one of the worst things I've seen put into python since then. I've used a wide range of frameworks (twisted, gevent, etc) as well as threads and even if async is a good solution (I don't think it is) it broke awscli for quite some time (through aiobotocore and related package dependencies). It's too late in the game for long-term breaks like that or any backward…
Yep, it's 2022 and gevent is still the only solution for async & high concurrency that Just Works with the entire ecosystem of Python libraries without code changes. There's definitely some compute overhead compared to async, but we save so much developer time having effortless concurrency and never being worried that, say, using a slow third-party API over the web will slow down other requests.