Show HN: 1M rows/s from Postgres to Python • Hacker News Board

Re: Show HN: 1M rows/s from Postgres to Python

#81

Post by nhumrich » Fri, Aug 05, 2016, 7:24 AM UTC

Thaxll wrote: post #21

1st1 wrote: post #17

Earlier quoted context omitted.

Python and nodejs applications are usually deployed in a multiprocess configuration (and Go apps without GOMAXPROCS). So in production, applications in all of those languages use all CPU cores. For benchmarks, we didn't want to complicate things with multi-process setup, the idea was to compare the raw performance of all drivers.

"usually deployed" where? Every Python app I've seen in Prod don't use multi process to bypass the limitation of GIL/#cores.

GIL only applies to multi-threaded, not multi-process. Multi-process is a great way to overcome the GIL

Re: Show HN: 1M rows/s from Postgres to Python

#82

Post by Fiahil » Fri, Aug 05, 2016, 7:33 AM UTC

We've been using Python (2.7) and Postgres, where I work, and I must say I really miss the async feature. It's really convenient to be able to parallelize and saturate I/O very quickly. Right now, we're using a combination of threads and processes without proper messaging, which leads to deadlocks and other nasty things. That's one thing I've been quite pleased with, when using Scala.

Re: Show HN: 1M rows/s from Postgres to Python

#83

Post by landmark3 » Fri, Aug 05, 2016, 8:24 AM UTC

time for a rollback uUber

Re: Show HN: 1M rows/s from Postgres to Python

#84

Post by amelius » Fri, Aug 05, 2016, 1:03 PM UTC

What about the complexity of the rows? For instance, how many columns per row?

Re: Show HN: 1M rows/s from Postgres to Python

#85

Post by RedCrowbar » Fri, Aug 05, 2016, 1:06 PM UTC

amelius wrote: post #84

What about the complexity of the rows? For instance, how many columns per row?

12 in the particular case of 1M rows/s. The details are available in the benchmark report: http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python...

Re: Show HN: 1M rows/s from Postgres to Python

#86

Post by pxlfkr » Fri, Aug 05, 2016, 1:07 PM UTC

Using Cython means you don't/can't use Pypy? Was that something you considered, e.g. with psycopg2cffi or a pure-python driver? It seems to be the defacto performant runtime for Python, and since performance is something you guys obviously care a lot about, I'm surprised it hasn't been mentioned in this thread.

Very excited to try out your code - thanks for sharing it.

Re: Show HN: 1M rows/s from Postgres to Python

#87

Post by RedCrowbar » Fri, Aug 05, 2016, 1:10 PM UTC

userbinator wrote: post #74

At 3.7GHz that's roughly 3700 clock cycles per row. With something that low, I'd be really interested to see a breakdown of how many cycles were spent doing what.

In the 1M benchmark roughly half is spent decoding rows (in [1]). The rest is spread over interpreter overhead, GC in particular.

[1] https://github.com/MagicStack/asyncpg/blob/master/asyncpg/pr...

Re: Show HN: 1M rows/s from Postgres to Python

#88

Post by RedCrowbar » Fri, Aug 05, 2016, 1:15 PM UTC

pxlfkr wrote: post #86

Using Cython means you don't/can't use Pypy? Was that something you considered, e.g. with psycopg2cffi or a pure-python driver? It seems to be the defacto performant runtime for Python, and since performance is something you guys obviously care a lot about, I'm surprised it hasn't been mentioned in this thread. Very excited to try out your code - thanks for sharing it.

A pure-Python implementation would probably never reach this level of performance, even under PyPy. Cython really makes it possible to basically drop down to C and write very efficient code.

That said, once PyPy supports 3.5, we will consider making a CFFI binding.

Re: Show HN: 1M rows/s from Postgres to Python

#89

Post by nhumrich » Fri, Aug 05, 2016, 1:16 PM UTC

pxlfkr wrote: post #86

Using Cython means you don't/can't use Pypy? Was that something you considered, e.g. with psycopg2cffi or a pure-python driver? It seems to be the defacto performant runtime for Python, and since performance is something you guys obviously care a lot about, I'm surprised it hasn't been mentioned in this thread. Very excited to try out your code - thanks for sharing it.

By using python asyncio, you can't use pypy because pypy doesn't support 3.5 yet (or probably for a while), so by going async, you are eliminating pypy right there.

Re: Show HN: 1M rows/s from Postgres to Python

#90

Post by 1st1 » Fri, Aug 05, 2016, 1:50 PM UTC

sametmax wrote: post #79

RedCrowbar wrote: post #49

Earlier quoted context omitted.

> I have no intention to stifle anything, I'm only raising the point that this seems like a really new driver and it's likely that it would take a while before it approaches the stability of psycopg2. That is, the stability of psycogp2 is not to be so lightly thrown in the trash. We in no way question the merits and the stability if psycopg2. We created asyncpg to solve the challenges we face in the development of Ed…

The tone used is indeed not the proper one, but while I'm a big asyncio advocate, I think it would be beneficial if you would split your project in 2, separating the IO from the rest (like hyper does https://github.com/Lukasa/hyper ). The rational being: - Python is not just async. It currently majoritarly sync. A lot of the community would benefit from your awesome project; - You'd have contrib from the async and sy…

> I think it would be beneficial if you would split your project in 2, separating the IO from the rest (like hyper does https://github.com/Lukasa/hyper).

I'd like to do that, but we don't have resources. If we had a couple capable devs volunteering on the GH to do this, I'd be glad to review patches and guide the development.