41–50 of 94 posts
would this work with django?
asyncpg was designed for asyncio (and async/await). So it can be used in any asyncio, Tornado (and soon Twisted) program. Django doesn't support asyncio, it's built for blocking IO model. So, unfortunately, it's not possible to use asyncpg with it.
Earlier quoted context omitted.
asyncpg was designed for asyncio (and async/await). So it can be used in any asyncio, Tornado (and soon Twisted) program. Django doesn't support asyncio, it's built for blocking IO model. So, unfortunately, it's not possible to use asyncpg with it.
But could be if the code is made to lock? ie, as if the async part not matter?
Does this library address any of the issues/concerns that Mike Bayer discussed in this earlier blog post? http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
the 1M row/sec part is easy to explain: > asyncpg extensively uses PostgreSQL prepared statements. This is an important optimization feature, as it allows to avoid repeated parsing, analysis, and planning of queries. Additionally, asyncpg caches the data I/O pipeline for each prepared statement. it would be great if psycopg2 did this, at least within the context of executemany(). You don't need async to use prepared…
Prepared statements were used in all benchmarks where supported. Please don't dismiss the results so easily. The source of performance increases here is not the use of prepared statements, but efficient implementation of the binary type I/O and the protocol itself.
> I'd be very wary of a new PG driver especially one that implements the wire protocol itself. > pg8000 does this, and also does prepared statements, and while it doesn't have the speed approach here, it also has serious problems supporting all the features and datatypes that psycopg2 does.
asyncpg supports 100% of pg_catalog types (we have a test for that.) Additionally, psycopg2 doesn't _really_ support all the features and datatypes and simply returns unparsed text in many cases. You have to write lots of typecasters to convince it otherwise.
> It's been around for years and reliably breaks on many Postgresql features (and Postgresql introduces new features like a firehose). Writing and maintaining a driver for Postgresql is not an easy task.
Please. Simply because something has been around is not the reason to stiffle new development and innovation.
Does this library address any of the issues/concerns that Mike Bayer discussed in this earlier blog post? http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
the 1M row/sec part is easy to explain: > asyncpg extensively uses PostgreSQL prepared statements. This is an important optimization feature, as it allows to avoid repeated parsing, analysis, and planning of queries. Additionally, asyncpg caches the data I/O pipeline for each prepared statement. it would be great if psycopg2 did this, at least within the context of executemany(). You don't need async to use prepared…
This simply isn't true for all use cases. If you have to manage 100-1000s of connections (some of each can be long-living sessions such as websockets) you have no other option but to use the async approach.
To minimize the number of connections to the DB you should use a decent caching layer (redis/memcached will do). Once you have that, and a use a connection pool, you'll be able to do a lot more with asyncio than with threads.
> The GIL doesn't block on IO so threads work just fine, and most applications are CRUD / transactional applications which means you can't spawn off several dozen queries within a transaction at the same time anyway, you need to wait for each one to complete serially regardless.
If you have 10 threads and relatively low server load - the threads will work just fine. However GIL will make things really slow under load.
Earlier quoted context omitted.
The protocol is documented in the official PostgreSQL documentation: https://www.postgresql.org/docs/current/static/protocol.html
I meant a doc explaining roughly the components at https://github.com/MagicStack/asyncpg/tree/master/asyncpg/pr... and how they interact with each other at a high level as it doesn't seem to have comments. Thanks for the link though, I'll have a look
Earlier quoted context omitted.
Python and nodejs applications are usually deployed in a multiprocess configuration (and Go apps without GOMAXPROCS). So in production, applications in all of those languages use all CPU cores. For benchmarks, we didn't want to complicate things with multi-process setup, the idea was to compare the raw performance of all drivers.
"usually deployed" where? Every Python app I've seen in Prod don't use multi process to bypass the limitation of GIL/#cores.
I run 80 servers with custom written daemons and it's always multi-process.
I just finished said infra at one of the worlds largest online betting sites, their legacy was multi-process and the new systems are multi-process.
This is really common
Earlier quoted context omitted.
the 1M row/sec part is easy to explain: > asyncpg extensively uses PostgreSQL prepared statements. This is an important optimization feature, as it allows to avoid repeated parsing, analysis, and planning of queries. Additionally, asyncpg caches the data I/O pipeline for each prepared statement. it would be great if psycopg2 did this, at least within the context of executemany(). You don't need async to use prepared…
> The TL;DR; for my blog post you mention is that async itself usually has little to do with speed when talking to a database, because the database is typically on the same network as your application, you're using typically no more than a few dozen connections per process (postgresql uses a process per connection, so using thousands of connections isn't feasible anyway), so the overhead of async and especially Pytho…
Everytime I talk about my blog post, I am super careful to qualify: in this case, "usually has little to do w/ speed" and other qualifications. The blog post as well. But everytime, I get a response, "but that's not always TRUE!" We agree. It is not always true. But it is usually true :).
> If you have to manage 100-1000s of connections
Which I said, usually you are not. Postgresql connections are expensive. You do not want 100s-1000s.
> some of each can be long-living sessions such as websockets
you use websockets to talk to your Postgresql database? Where did I say I was talking about HTTP web services ?
> However GIL will make things really slow under load.
not IO load with a hundred threads or so. Thousands, sure. Otherwise no.
Earlier quoted context omitted.
the 1M row/sec part is easy to explain: > asyncpg extensively uses PostgreSQL prepared statements. This is an important optimization feature, as it allows to avoid repeated parsing, analysis, and planning of queries. Additionally, asyncpg caches the data I/O pipeline for each prepared statement. it would be great if psycopg2 did this, at least within the context of executemany(). You don't need async to use prepared…
> the 1M row/sec part is easy to explain: Prepared statements were used in all benchmarks where supported. Please don't dismiss the results so easily. The source of performance increases here is not the use of prepared statements, but efficient implementation of the binary type I/O and the protocol itself. > I'd be very wary of a new PG driver especially one that implements the wire protocol itself. > pg8000 does thi…
Does your driver seamlessly handle reading and writing of three-dimensional arrays of JSONB structures, and arrays of custom enumerated types ?
edit:
> but efficient implementation of the binary type I/O and the protocol itself.
that's great too! you don't need async for that either.
I guess I am frustrated that you folks put all this work into a great driver and then locked it in the largely useless dungeon of asyncio, where all the libraries are totally useless to those of us who don't think node.js is the programming model for all cases.
Earlier quoted context omitted.
> the 1M row/sec part is easy to explain: Prepared statements were used in all benchmarks where supported. Please don't dismiss the results so easily. The source of performance increases here is not the use of prepared statements, but efficient implementation of the binary type I/O and the protocol itself. > I'd be very wary of a new PG driver especially one that implements the wire protocol itself. > pg8000 does thi…
I have no intention to stifle anything, I'm only raising the point that this seems like a really new driver and it's likely that it would take a while before it approaches the stability of psycopg2. That is, the stability of psycogp2 is not to be so lightly thrown in the trash. Does your driver seamlessly handle reading and writing of three-dimensional arrays of JSONB structures, and arrays of custom enumerated types…
We in no way question the merits and the stability if psycopg2. We created asyncpg to solve the challenges we face in the development of EdgeDB, it's not just a toy project for fun.
> Does your driver seamlessly handle reading and writing of three-dimensional arrays of JSONB structures, and arrays of custom enumerated types ?
Yes it does.
> edit:
> > but efficient implementation of the binary type I/O and the protocol itself.
> that's great too! you don't need async for that either.
> I guess I am frustrated that you folks put all this work into a great driver and then locked it in the largely useless dungeon of asyncio, where all the libraries are totally useless to those of us who don't think node.js is the programming model for all cases.
I'm sorry to hear that. Nobody forces you to use asyncio or asyncpg. But please don't try to claim that something is a "useless dungeon" simply because you disagree.
Earlier quoted context omitted.
> The TL;DR; for my blog post you mention is that async itself usually has little to do with speed when talking to a database, because the database is typically on the same network as your application, you're using typically no more than a few dozen connections per process (postgresql uses a process per connection, so using thousands of connections isn't feasible anyway), so the overhead of async and especially Pytho…
> This simply isn't true for all use cases. Everytime I talk about my blog post, I am super careful to qualify: in this case, " usually has little to do w/ speed" and other qualifications. The blog post as well. But everytime, I get a response, "but that's not always TRUE!" We agree. It is not always true. But it is usually true :). > If you have to manage 100-1000s of connections Which I said, usually you are not. P…
> Which I said, usually you are not. Postgresql connections are expensive. You do not want 100s-1000s.
I was talking about a micro-service that has to support a lot of connections and work with a DB.
>> some of each can be long-living sessions such as websockets
> you use websockets to talk to your Postgresql database? Where did I say I was talking about HTTP web services ?
So you were talking about programs that don't communicate with the outside world except to a DB?