Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: 1M rows/s from Postgres to Python

magic.io

Post reply on HN
Print view View original

21–30 of 94 posts

Re: Show HN: 1M rows/s from Postgres to Python

#21

Post by Thaxll »

1st1 wrote: post #17
jargino wrote: post #15

> For fairness, all tests were run in a single-thread (GOMAXPROCS=1 for Go code) in async mode. Isn't this just hobbling Go so you can beat it? Doesn't seem like something done "for fairness" at all.

Python and nodejs applications are usually deployed in a multiprocess configuration (and Go apps without GOMAXPROCS). So in production, applications in all of those languages use all CPU cores. For benchmarks, we didn't want to complicate things with multi-process setup, the idea was to compare the raw performance of all drivers.

"usually deployed" where? Every Python app I've seen in Prod don't use multi process to bypass the limitation of GIL/#cores.

Re: Show HN: 1M rows/s from Postgres to Python

#24

Post by 1st1 »

tjpaudio wrote: post #11

This is really awesome, I'm going to give it a try in my latest project. Has there been any apps/sites with large amounts of i/o running this yet? Curious how reliable it is at this stage.

We've open sourced asyncpg just a couple of weeks ago, and right now it has a beta status. Not that we're aware of any bugs in it or plan to add a lot of new features. So I guess you should give it a try, carefully ;)

Re: Show HN: 1M rows/s from Postgres to Python

#26

Post by RedCrowbar »

Keats wrote: post #25
1st1 wrote: post #2

asyncpg was developed by @redcrowbar and me. Ask us anything ;)

I'd love to read the docs for the protocol itself, is it planned? Do you have a link explaining it otherwise? Great work!

The protocol is documented in the official PostgreSQL documentation: https://www.postgresql.org/docs/current/static/protocol.html

Re: Show HN: 1M rows/s from Postgres to Python

#27

Post by riquito »

Thaxll wrote: post #21
1st1 wrote: post #17

Earlier quoted context omitted.

Python and nodejs applications are usually deployed in a multiprocess configuration (and Go apps without GOMAXPROCS). So in production, applications in all of those languages use all CPU cores. For benchmarks, we didn't want to complicate things with multi-process setup, the idea was to compare the raw performance of all drivers.

"usually deployed" where? Every Python app I've seen in Prod don't use multi process to bypass the limitation of GIL/#cores.

supervisord

Re: Show HN: 1M rows/s from Postgres to Python

#29

Post by [deleted] »

1st1 wrote: post #17
jargino wrote: post #15

> For fairness, all tests were run in a single-thread (GOMAXPROCS=1 for Go code) in async mode. Isn't this just hobbling Go so you can beat it? Doesn't seem like something done "for fairness" at all.

Python and nodejs applications are usually deployed in a multiprocess configuration (and Go apps without GOMAXPROCS). So in production, applications in all of those languages use all CPU cores. For benchmarks, we didn't want to complicate things with multi-process setup, the idea was to compare the raw performance of all drivers.

[deleted]

Re: Show HN: 1M rows/s from Postgres to Python

#30

Post by 1st1 »

bedros wrote: post #23

would this work with django?

asyncpg was designed for asyncio (and async/await). So it can be used in any asyncio, Tornado (and soon Twisted) program.

Django doesn't support asyncio, it's built for blocking IO model. So, unfortunately, it's not possible to use asyncpg with it.

Post reply on HN