Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Write Fast Apps Using Async Python 3.6 and Redis

eng.paxos.com

Post reply on HN
Print view View original

101–110 of 134 posts

Re: Write Fast Apps Using Async Python 3.6 and Redis

#101

Post by siscia »

Quite related, but if you want to use Redis as a SQL database I wrote an extension to do just that: https://github.com/RedBeardLab/rediSQL

It is a relative thin layer of rust code between the Redis module interface and SQLite.

At the moment you can simply execute statements but any suggestion and feature request is very welcome.

Yes, it is possible to do join, to use the LIKE operator and pretty much everything that SQLite gives you.

It is a multi-thread module, which means that it does NOT block the main redis thread and perform quite well. On my machine I achieved 50.000 inserts per seconds for the in memory database.

If you have any question feel free to ask here or to open issues and pull request in the main repo.

:)

Re: Write Fast Apps Using Async Python 3.6 and Redis

#102

Post by sandGorgon »

tomchristie wrote: post #93
zebra9978 wrote: post #86

Earlier quoted context omitted.

any particular reason you are using BSD license ? With all due respect, this does not cover a patent grant like the Apache license and could be a poison pill for companies to adopt.

I just had a quick scan over the licenses of other projects used for server side projects. Projects using BSD/MIT include Node, Go, Rails, Django, and Flask. I'm happy with the choice.

Sorry to interject - but that's not completely true. Go comes with a separate patent disclaimer.

https://golang.org/PATENTS

And IMHO nodejs is not a standard BSD license and comes with patent grant. That discussion went on for a year in the TSC . https://github.com/nodejs/node/blob/master/LICENSE

In general, this stuff is not always evident. But the BSD license by itself is not as good as Apache.

While in doubt, use Apache !

P.S. fyi, doing this later is super heavy-duty hard.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#103

Post by cies »

> Write Fast Apps Using Async Python

When working with Python and Ruby I find 80ms responses acceptable. In very optimized situations (no framework) this can do down to 20ms.

Now I've used some Haskell, OCaml and Go and I have learned that they can typically respond in In both cases this includes querying the db several times (db queries usually take less then a millisecond, Redis shall be quite similar to the extend that it does not change outcome).

I've come to the conclusion that --considering other languages-- speed is not to be found in Python and Ruby.

Apart from the speed story there's also resource consumption, and in that game it is only compiled languages that truly compete.

Last point: give the point I make above and that nowadays "the web is the UI", I believe that languages for hi-perf application development should: compile to native and compile to JS. Candidates: OCaml/Reason (BuckleScript), Haskell (GHCJS), PureScript (ps-native), [please add if I forgot any]

Re: Write Fast Apps Using Async Python 3.6 and Redis

#104

Post by Dowwie »

zzzeek wrote: post #22

> we make heavy use of asyncio because it’s more performant more performant than....what exactly? If I need to load 1000 rows from a database and splash them on a webpage, will my response time go from the 300ms it takes without asyncio to something "more performant", like 50ms? Answer: no. async only gives you throughput, it has nothing to do with "faster" as far as the Python interpreter / GIL / anything like that.…

You are the hero we need, Mike

Re: Write Fast Apps Using Async Python 3.6 and Redis

#105

Post by anentropic »

zzzeek wrote: post #22

> we make heavy use of asyncio because it’s more performant more performant than....what exactly? If I need to load 1000 rows from a database and splash them on a webpage, will my response time go from the 300ms it takes without asyncio to something "more performant", like 50ms? Answer: no. async only gives you throughput, it has nothing to do with "faster" as far as the Python interpreter / GIL / anything like that.…

If you have to make several requests to db backend to fulfil one response then potentially asyncio allows you to make them in parallel rather than in series. Reducing latency of your response.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#106

Post by coldtea »

zzzeek wrote: post #94
makmanalp wrote: post #87

Earlier quoted context omitted.

> this is already going to be a heavy web request with multiple web service calls Sure, but it can now be a less heavy web request! ¯\_(ツ)_/¯ > a request typically has a single transaction going out to the database The fact of the matter is, as applications develop, become richer, and grow larger, it becomes less and less uncommon to have more than one query per page. Especially in the context of larger organizations…

> it becomes less and less uncommon to have more than one query per page. I said transaction, not query. A database transaction is on a single connection at a time and queries are performed via the transaction serially.

>I said transaction, not query. A database transaction is on a single connection at a time and queries are performed via the transaction serially.

Unless you're doing e-commerce or banking sites, that's far less common that non-transaction requests.

Re: Write Fast Apps Using Async Python 3.6 and Redis

#107

Post by hawski »

matthewaveryusa wrote: post #67
zzzeek wrote: post #22

> we make heavy use of asyncio because it’s more performant more performant than....what exactly? If I need to load 1000 rows from a database and splash them on a webpage, will my response time go from the 300ms it takes without asyncio to something "more performant", like 50ms? Answer: no. async only gives you throughput, it has nothing to do with "faster" as far as the Python interpreter / GIL / anything like that.…

It buys you the stack size of each thread which only matters if you have a stupid amount of connections. In this article[1] the author makes a comparison between the 2 models and 7000 concurrent users will chew up 450MB of stack space. Of course this is adjustable. [1] http://byteworm.com/evidence-based-research/2017/03/04/compa...

On most Linux systems stack is allocated with mmap with overcommiting. Until first write all those pages will share same zeroed page AFAIK. Then only overwritten pages will be allocated.

Am I wrong?

Re: Write Fast Apps Using Async Python 3.6 and Redis

#108

Post by floatboth »

twistedpair wrote: post #65
hasenj wrote: post #62

If you want performance don't use Python.

Sadly true. Python is great for scripting, but "high performance Python" is frequently a challenge better suited to other tools.

"High performance Python" is usually done by "offloading literally everything to native extensions" :D

Re: Write Fast Apps Using Async Python 3.6 and Redis

#109

Post by zzzeek »

kazagistar wrote: post #78
zzzeek wrote: post #60

Earlier quoted context omitted.

So on that topic, a request typically has a single transaction going out to the database so within the scope of the request, has to perform its steps in serial in any case. If it needs to make several requests to web services that aren't dependent on each other, that's an area where you can get into stacking them with some kind of concurrency construct (I'd pass it into a greenlet oriented worker pool). but this is a…

> a request typically has a single transaction going out to the database Its typical because people are still in a "single thread single transaction ORM crud" model of thinking. "Its linear because thats how it is"?

if you're using an ACID kind of database then yes, that's how it is :)

Re: Write Fast Apps Using Async Python 3.6 and Redis

#110

Post by zzzeek »

coldtea wrote: post #106
zzzeek wrote: post #94

Earlier quoted context omitted.

> it becomes less and less uncommon to have more than one query per page. I said transaction, not query. A database transaction is on a single connection at a time and queries are performed via the transaction serially.

> I said transaction, not query. A database transaction is on a single connection at a time and queries are performed via the transaction serially. Unless you're doing e-commerce or banking sites, that's far less common that non-transaction requests.

unless you're using MyISAM or something like that, all your queries are in transactions.

edit: also, I'd challenge you to prove that for a web request that needs to make ten read queries to a relational database, from Python, that you can get better performance by opening up ten separate database connections (or from a pool) and running one query in each, bundled into the async construct of your choice and then merging them all back into your response, vs. just running ten queries on a single connection in serial. Assume these are not slow reporting-style transactions, just the usual "load the users full name, load the current status, load the user's current items", etc., small queries common in a web request that is looking for a very fast response with ten SQL queries.

Note that at the very least, it means your web application needs to use ten times as many database connections for a given set of load. In database-land that's more or less crazy.

Post reply on HN