This is really awesome, I'm going to give it a try in my latest project. Has there been any apps/sites with large amounts of i/o running this yet? Curious how reliable it is at this stage.
Show HN: 1M rows/s from Postgres to Python
11–20 of 94 posts
Re: Show HN: 1M rows/s from Postgres to Python
#12would be cool to also make jmh tests against jdbc and async pg on java.
actually what i also found is that python wasn't slow in these cases but it was slow when you do map operations on the lists and reoder the data or convert it to other structures.
Re: Show HN: 1M rows/s from Postgres to Python
#13[deleted]
Re: Show HN: 1M rows/s from Postgres to Python
#14asyncpg was developed by @redcrowbar and me. Ask us anything ;)
How can I use it with SQLAlchemy instead of psycopg2?
Re: Show HN: 1M rows/s from Postgres to Python
#15> For fairness, all tests were run in a single-thread (GOMAXPROCS=1 for Go code) in async mode.
Isn't this just hobbling Go so you can beat it? Doesn't seem like something done "for fairness" at all.
Re: Show HN: 1M rows/s from Postgres to Python
#16asyncpg was developed by @redcrowbar and me. Ask us anything ;)
Do you have any numbers on queries that return millions of rows? If it's still fast, asyncpg could be nice for some ETL-type workloads.
Re: Show HN: 1M rows/s from Postgres to Python
#17> For fairness, all tests were run in a single-thread (GOMAXPROCS=1 for Go code) in async mode. Isn't this just hobbling Go so you can beat it? Doesn't seem like something done "for fairness" at all.
Python and nodejs applications are usually deployed in a multiprocess configuration (and Go apps without GOMAXPROCS). So in production, applications in all of those languages use all CPU cores.
For benchmarks, we didn't want to complicate things with multi-process setup, the idea was to compare the raw performance of all drivers.
Re: Show HN: 1M rows/s from Postgres to Python
#18would be cool to also make jmh tests against jdbc and async pg on java. actually what i also found is that python wasn't slow in these cases but it was slow when you do map operations on the lists and reoder the data or convert it to other structures.
> would be cool to also make jmh tests against jdbc and async pg on java.
Pull requests to https://github.com/MagicStack/pgbench are welcome :-)
Re: Show HN: 1M rows/s from Postgres to Python
#19asyncpg was developed by @redcrowbar and me. Ask us anything ;)
Do you have any numbers on queries that return millions of rows? If it's still fast, asyncpg could be nice for some ETL-type workloads.
We haven't benchmarked this specific use case. However, cursors in asyncpg do data prefetching (the exact number of prefetched rows is configurable), so I expect that asynchronous iteration should be pretty fast (and still faster than psycopg2).