Asynchronous Python and Databases
techspot.zzzeek.org
Asynchronous Python and Databases
1–10 of 76 posts
Re: Asynchronous Python and Databases
#2> ...the speed of Python is not nearly as fast as your database, when dealing in terms of standard CRUD-style applications...
The blanket assumptions that most developers are building "standard CRUD-style applications" and that the database is never a bottleneck make it difficult for me to take this post seriously. Sure, if you're running a simple to-do list app, you probably won't need asynchronous I/O, but in my experience, real-world business apps actually tend to use fairly complex database logic. And sure, there are ways of speeding up your app that don't involve touching the Python side (caching using Redis, denormalizing, etc), but they are a lot harder to implement than asynchronous I/O. Why give up on some free performance for many types of applications?
Also, the author mentions and then completely dismisses PyPy. Why? I've used PyPy on a production web app, and I can say for certain that from that point, the Python code completely ceased being the bottleneck at any point--the majority of the time spent was in the database. Again, I could have implemented better caching and restructured my schema, but at that point, dropping in asyncio would have been way easier and allowed the app to scale with much less effort.
There is definitely some validity to the post, but the author makes blanket assumptions that drastically weaken his argument.
Re: Asynchronous Python and Databases
#3> ...the speed of Python is not nearly as fast as your database, when dealing in terms of standard CRUD-style applications... The blanket assumptions that most developers are building "standard CRUD-style applications" and that the database is never a bottleneck make it difficult for me to take this post seriously. Sure, if you're running a simple to-do list app, you probably won't need asynchronous I/O, but in my ex…
Well, I've posted my benchmarks. Where are yours ?
Re: Asynchronous Python and Databases
#4> ...the speed of Python is not nearly as fast as your database, when dealing in terms of standard CRUD-style applications... The blanket assumptions that most developers are building "standard CRUD-style applications" and that the database is never a bottleneck make it difficult for me to take this post seriously. Sure, if you're running a simple to-do list app, you probably won't need asynchronous I/O, but in my ex…
Well, I've posted my benchmarks. Where are yours ?
I ran benchmarks for the app I mentioned. I don't have them on me because I no longer work for the company, but the database was the bottleneck on every request once we switched to PyPy. Again, I'm not saying your benchmarks are invalid, but the assumptions you make don't hold for all (or probably even the majority) of business web apps.
Re: Asynchronous Python and Databases
#5Earlier quoted context omitted.
Well, I've posted my benchmarks. Where are yours ?
I ran benchmarks for the app I mentioned. I don't have them on me because I no longer work for the company, but the database was the bottleneck on every request once we switched to PyPy. Again, I'm not saying your benchmarks are invalid, but the assumptions you make don't hold for all (or probably even the majority) of business web apps.
Anecdotally, I've seen quite a few Django apps using gevent "just because". If you don't need it, you're just incurring additional overhead for no gain (which is what the author is saying).
There are legitimate usage cases for async patterns, but I have seen them used even in situations where they made things slower.
Re: Asynchronous Python and Databases
#6Earlier quoted context omitted.
Well, I've posted my benchmarks. Where are yours ?
I ran benchmarks for the app I mentioned. I don't have them on me because I no longer work for the company, but the database was the bottleneck on every request once we switched to PyPy. Again, I'm not saying your benchmarks are invalid, but the assumptions you make don't hold for all (or probably even the majority) of business web apps.
My post is aimed first and foremost at the Openstack community. Openstack is a large series of CRUD-based applications that don't run any stored procedures or business logic on the database. As to whether or not the "majority" of business apps are stored procedure oriented is very questionable, I've worked for years in finance and while I worked on stored procedure applications for sure, most apps were definitely just CRUD-based apps. The vast majority of apps we see for example in Django or Pyramid are most definitely CRUD based apps.
Also, I use Pypy a lot, and at best I get about a 40-50% speedup. If we factor that into the profiling where I'm showing IO taking up about 5% of the time, that still produces an application that is very much CPU bound.
If there's any point to take from the post, its the historical underpinnings of non-blocking IO, that we use it to poll a bunch of sleeping connections, where we have hundreds or thousands of them. It was never intended to be used on a small set of database connections and I think even if the database calls are slow, it will be very seldom that you'd see asynchronous context switching handling concurrency better than native threads.
Re: Asynchronous Python and Databases
#7I just want to note that asynchronous programming is analogous to cooperative multitasking, as used in the Windows 3.1 era. It seems, if we value low latency, that we should not pursue that route for the long term, or we should be very cautious about it. For tasks that are not purely I/O bound, its use is questionable.
Re: Asynchronous Python and Databases
#8I just want to note that asynchronous programming is analogous to cooperative multitasking, as used in the Windows 3.1 era. It seems, if we value low latency, that we should not pursue that route for the long term, or we should be very cautious about it. For tasks that are not purely I/O bound, its use is questionable.
[deleted]
Re: Asynchronous Python and Databases
#9If you need that much performance out of Python, it's probably time to switch to Go. With Python, you still have the Global Interpreter Lock, even in PyPy. Multiple CPUs, which you probably have available, don't help.
Re: Asynchronous Python and Databases
#10If you need that much performance out of Python, it's probably time to switch to Go. With Python, you still have the Global Interpreter Lock, even in PyPy. Multiple CPUs, which you probably have available, don't help.
This isn't an option when you have a large legacy code base in Python and you aren't the size of Dropbox.