Cool! Wish there would be a Django wrapper, but then realised it is designed for asyncio
Show HN: 1M rows/s from Postgres to Python
71–80 of 94 posts
Re: Show HN: 1M rows/s from Postgres to Python
#72Would this work/be binary-compatible with Amazon Red Shift?
Should be. AFAIK psycopg2 works with Redshift just fine, so should asyncpg.
The only thing I'm not sure about is what kind of authentication mode Redshift is using. asyncpg doesn't support all of them, but that's easy to fix.
Re: Show HN: 1M rows/s from Postgres to Python
#73Would this work/be binary-compatible with Amazon Red Shift?
Haven't tried, but, AFAIK, redshift speaks the standard PostgreSQL protocol, so asyncpg should work.
Re: Show HN: 1M rows/s from Postgres to Python
#74At 3.7GHz that's roughly 3700 clock cycles per row. With something that low, I'd be really interested to see a breakdown of how many cycles were spent doing what.
Re: Show HN: 1M rows/s from Postgres to Python
#75How do you develop your Cython code? Do you use PyCharm? Did you start off from pure Python code? How easy is debugging and refactoring?
Re: Show HN: 1M rows/s from Postgres to Python
#76How do you develop your Cython code? Do you use PyCharm? Did you start off from pure Python code? How easy is debugging and refactoring?
I use SublimeText with a Cython highlighter; @redcrowbar uses Atom. I started coding uvloop & asyncpg directly in Cython, basically from scratch.
Debugging & refactoring are pretty much as easy as for pure Python code (which is a huge win over C). Sometimes when you're writing low-level code you can get a segfault, in which case lldb (or gdb) help you understand what's going on relatively quickly.
Re: Show HN: 1M rows/s from Postgres to Python
#77How do you develop your Cython code? Do you use PyCharm? Did you start off from pure Python code? How easy is debugging and refactoring?
I use SublimeText with a Cython highlighter; @redcrowbar uses Atom. I started coding uvloop & asyncpg directly in Cython, basically from scratch. Debugging & refactoring are pretty much as easy as for pure Python code (which is a huge win over C). Sometimes when you're writing low-level code you can get a segfault, in which case lldb (or gdb) help you understand what's going on relatively quickly.
So do you use cygdb to debug Cython code or directly debug the generated C-API code?
Re: Show HN: 1M rows/s from Postgres to Python
#78Earlier quoted context omitted.
I use SublimeText with a Cython highlighter; @redcrowbar uses Atom. I started coding uvloop & asyncpg directly in Cython, basically from scratch. Debugging & refactoring are pretty much as easy as for pure Python code (which is a huge win over C). Sometimes when you're writing low-level code you can get a segfault, in which case lldb (or gdb) help you understand what's going on relatively quickly.
So do you use cygdb to debug Cython code or directly debug the generated C-API code?
I usually debug the generated C code directly (usually it's quite easy to understand the bug once you see where the program crashes.)
Re: Show HN: 1M rows/s from Postgres to Python
#79Earlier quoted context omitted.
I have no intention to stifle anything, I'm only raising the point that this seems like a really new driver and it's likely that it would take a while before it approaches the stability of psycopg2. That is, the stability of psycogp2 is not to be so lightly thrown in the trash. Does your driver seamlessly handle reading and writing of three-dimensional arrays of JSONB structures, and arrays of custom enumerated types…
> I have no intention to stifle anything, I'm only raising the point that this seems like a really new driver and it's likely that it would take a while before it approaches the stability of psycopg2. That is, the stability of psycogp2 is not to be so lightly thrown in the trash. We in no way question the merits and the stability if psycopg2. We created asyncpg to solve the challenges we face in the development of Ed…
The tone used is indeed not the proper one, but while I'm a big asyncio advocate, I think it would be beneficial if you would split your project in 2, separating the IO from the rest (like hyper does https://github.com/Lukasa/hyper).
The rational being:
- Python is not just async. It currently majoritarly sync. A lot of the community would benefit from your awesome project; - You'd have contrib from the async and sync community; - async may sometime not be worth it for db (http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an... - some people will prefer to use threads or gevent; - you will see compat plugin created for current ORM;
That aside, I will use your project as is, since I'm deep in asyncio these days. Thank you.
Re: Show HN: 1M rows/s from Postgres to Python
#80Earlier quoted context omitted.
It might even be easier than writing an adapter if you just use SQLAlchemy Core's DSL as the generative mechanism for a SQL statement that is sent to asyncpg.
There appears to be an adapter for Core already: https://github.com/CanopyTax/asyncpgsa
Author of asyncpgsa here, I am in the process of writing that, let me know if you have questions.