361–364 of 364 posts
https://github.com/nDmitry/web-benchmarks
Long story short - asyncio is twice as fast... (results are at the bottom of the readme).
Async python is faster when you use it for running parallel tasks. In this benchmark, you are running a single database request per query, so there is no advantage to being asynchronous: a pool of processes will scale just as well (but it will use more memory). The point of async is that it lets you easily make a Postgres query, AND an HTTP query, AND a redis query in parallel.
Couldn’t threads handle that use case?
Earlier quoted context omitted.
On top of that, the author uses aiopg rather than asyncpg[1] for the async database operations, even though asyncpg is (allegedly) a whole lot faster. 1. https://github.com/MagicStack/asyncpg
asyncpg is not scalable. It can only do "session pooling" because it needs advisory_locks, listen/notify, which will end up needing a lot of Postgresql connections.
Earlier quoted context omitted.
asyncpg is not scalable. It can only do "session pooling" because it needs advisory_locks, listen/notify, which will end up needing a lot of Postgresql connections.
Can you share more information on this (articles, etc)?
1. One Postgresql connection is a forked process and has memory overhead (4MB iirc) + context switching.
2. A connection can only execute 1 concurrent query (no multiplexing).
3. Asyncpg to be fast, uses the features that I mentioned in my parent post. Those can only be used in Session Pooling https://www.pgbouncer.org/features.html.
The whole point of async is to some other work while waiting for a query (ex a different query).
If you have 10 servers with 16 cores, each vcore has 1 python process, each python process doing 10 simultaneous queries. 10 * 16 * 10 = 1600 opened connections.
The best way IMHO: Is to use autocommit connections. This way your transactions execute in 1 RPC. You can keep multiple connections opened with very light CPU and pooling is best.
I've done 20K short lived queries/second from 1 process with only ~20 connections opened in Postgresql (using Pgbouncer statement pooling).