Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asynchronous Python and Databases

techspot.zzzeek.org

Post reply on HN
Print view View original

21–30 of 76 posts

Re: Asynchronous Python and Databases

#21

Post by temuze »

zzzeek wrote: post #15
temuze wrote: post #14

If you have a Python CRUD app, there are many steps you should take to speed up the app before parallelizing queries. The first couple that come to mind: - Caching. The 90-10 rule applies to most CRUD apps. Use varnish to get rid of some requests before they hit your webserver or make a redis/memcache LRU layer to prevent queries from hitting your database. - Shifting more work away from Python to the database. A sur…

Why don't I put caching on my threaded app and leave it at that? Async gets me nothing.

Memory perhaps? Presumably, the threads would be a little heavier.

I would be curious to see how much memory each of your test cases used.

However, if you have memory problems in your webserver, it might be worth just spinning up more webservers...

Re: Asynchronous Python and Databases

#22

Post by detaro »

temuze wrote: post #14

If you have a Python CRUD app, there are many steps you should take to speed up the app before parallelizing queries. The first couple that come to mind: - Caching. The 90-10 rule applies to most CRUD apps. Use varnish to get rid of some requests before they hit your webserver or make a redis/memcache LRU layer to prevent queries from hitting your database. - Shifting more work away from Python to the database. A sur…

Really? write Cython modules before parallelizing?

Re: Asynchronous Python and Databases

#23

Post by zzzeek »

patronagezero wrote: post #18
zzzeek wrote: post #3

Earlier quoted context omitted.

Well, I've posted my benchmarks. Where are yours ?

Benchmarks are snake oil when not being given the ability to reproduce them.

Download my suite and run them! Show me asyncio beating out threads in some database-centric scenario. I was really hoping to see that happen in some scenario or another.

Re: Asynchronous Python and Databases

#24

Post by robbles »

temuze wrote: post #14

If you have a Python CRUD app, there are many steps you should take to speed up the app before parallelizing queries. The first couple that come to mind: - Caching. The 90-10 rule applies to most CRUD apps. Use varnish to get rid of some requests before they hit your webserver or make a redis/memcache LRU layer to prevent queries from hitting your database. - Shifting more work away from Python to the database. A sur…

> a surprising number of apps don't do pagination properly

Could you elaborate on that? What's "properly"? Do you mean that they don't do it at all, or they do it in memory instead of in an indexed query? Or is there a technique here that I'm missing?

Re: Asynchronous Python and Databases

#25

Post by temuze »

detaro wrote: post #22
temuze wrote: post #14

If you have a Python CRUD app, there are many steps you should take to speed up the app before parallelizing queries. The first couple that come to mind: - Caching. The 90-10 rule applies to most CRUD apps. Use varnish to get rid of some requests before they hit your webserver or make a redis/memcache LRU layer to prevent queries from hitting your database. - Shifting more work away from Python to the database. A sur…

Really? write Cython modules before parallelizing?

Fair enough - probably not in an IO bound webserver :)

Re: Asynchronous Python and Databases

#26

Post by temuze »

robbles wrote: post #24
temuze wrote: post #14

If you have a Python CRUD app, there are many steps you should take to speed up the app before parallelizing queries. The first couple that come to mind: - Caching. The 90-10 rule applies to most CRUD apps. Use varnish to get rid of some requests before they hit your webserver or make a redis/memcache LRU layer to prevent queries from hitting your database. - Shifting more work away from Python to the database. A sur…

> a surprising number of apps don't do pagination properly Could you elaborate on that? What's "properly"? Do you mean that they don't do it at all, or they do it in memory instead of in an indexed query? Or is there a technique here that I'm missing?

A surprising number of people either make N queries because they don't what their code is doing (really easy for a beginner to do with Django models when not using prefetch) or, more commonly, they make queries that get N rows back.

You almost never need N rows. N rows are bad because Python has to parse them. Odds are, there's a way to get a constant number of rows back for every query in every view of your webapp.

Re: Asynchronous Python and Databases

#27

Post by threeseed »

> For database code, you have exactly one technique to use in order to assure correct concurrency, and that is by using ACID-oriented constructs and techniques.

Actually no. And in the transfer example he listed it isn't even how banks do it in the real world. Mostly they use eventual consistency. And there are alternatives to ACID albeit far less simple. You can use something like Zookeeper to handle transactions. Or if you are using an eventually consistent database e.g. Cassandra simply set the quorum such that you are querying all nodes.

Re: Asynchronous Python and Databases

#28

Post by threeseed »

zik wrote: post #13
temuze wrote: post #10

Earlier quoted context omitted.

This isn't an option when you have a large legacy code base in Python and you aren't the size of Dropbox.

If you're trying to convert a large legacy code base to async I/O you're up for a huge amount of work anyway.

Sure. But you aren't likely to be changing core business logic.

And believe me the cost of retraining, recoding and retesting all of your code is going to be significantly more expensive than buying better hardware.

Re: Asynchronous Python and Databases

#29

Post by PythonicAlpha »

amelius wrote: post #7

I just want to note that asynchronous programming is analogous to cooperative multitasking, as used in the Windows 3.1 era. It seems, if we value low latency, that we should not pursue that route for the long term, or we should be very cautious about it. For tasks that are not purely I/O bound, its use is questionable.

I disagree.

It is not analogous to cooperative multitasking, since Windows 3.1 dealt with different applications -- but asynchronous programming is targeting only a single (server) application.

I also would add, that asynchronous implemented server applications are doing oftentimes better than systems that primarily rely on threading.

Re: Asynchronous Python and Databases

#30

Post by bayesianhorse »

Asyncio isn't a good fit for classic web frameworks, like django. These applications connect to an excellent, fast and local database, maybe a caching server, and that's it. Their Http requests are designed to be quick and stateless.

A modern application may be designed very differently. Websockets almost require some kind of asynchronous concurrency, especially beyond simple push-notifications. Talking to remote databases, micro-services or even big-data frameworks is a very different ball game: Latency and processing time can quickly add up, making asyncio concurrency more attractive.

Finally, user input is very asynchronous and slow. Asyncio offers you to do stuff like "command = yield from interact(); if command == "start": ...", both in websockets and GUI frameworks like kivy.

Post reply on HN