Live data from Hacker News

Making 1M requests with Python-aiohttp

pawelmhm.github.io

31–40 of 84 posts

Re: Making 1M requests with Python-aiohttp

#31

"Everyone knows that asynchronous code performs better when applied to network operations" Ummm that seems a bit far reaching.

It depends on what "network operations" you are trying to do.

For high-concurrency purposes, asynchronous programming is far more scalable (see: epoll/kqueue + state machines).

For high-throughput, low-concurrency operations, it doesn't matter as much.

Re: Making 1M requests with Python-aiohttp

#32

I really keep wishing that there would be benchmark comparisons of asyncio/aiohttp with gevent/python2 . Performance would be a killer reason to migrate immediately to Py3. What I suspect though is that asyncio is not all that better than gevent. Can someone correct me on this?

Is there anything inherent to Python3 that is slower than Python2? Or is it just some of the performant packages still have not been ported to Python3?

Ironically enough unicode takes its toll on performance. This has mostly been made up though from 3.4 onwards.

Re: Making 1M requests with Python-aiohttp

#33
post #23

1,000,000 requests in 52 minutes is just 320 req/sec. Am I missing something? What's so amazing about this? I just deployed some production feed that serves at 1955 requests/second on a cheap VPS in freaking PHP, one of the slowest languages out there.

> Am I missing something? What's so amazing about this? The article is not about testing performance of a web server, but showcasing performance differences between synchronous and asynchronous code using asyncio. So, not about serving requests, but consuming.

Then he should change the title.

Re: Making 1M requests with Python-aiohttp

#34

Earlier quoted context omitted.

Is there anything inherent to Python3 that is slower than Python2? Or is it just some of the performant packages still have not been ported to Python3?

i keep looking for a reason to switch to python 3 and cant find one. Plus if I want to use the cool stuff in Pypy.. then I better not ! overall - very less reason to consider Py3 at all. Performance would have been one - if there were a comparison between gevent and asyncio.

>i keep looking for a reason to switch to python 3 and cant find one.

Unicode? Not having to deal with encoding all over the place has been well worth switch to Python 3. If performance is a a huge issue, I honestly don't know why you would stay on Python (regardless of version)

I wouldn't want to switch back to Python 2.7 is I can avoid it. There's honestly no reason not to go with 3.4 or 3.5 at this point, unless you happen to have a large Python 2 code base.

Re: Making 1M requests with Python-aiohttp

#35
post #16

I have a library for doing coordinated async IO in python that addresses some of the scheduling and resource contention issues hinted out in the later part of this post. It's called cellulario in reference to containing async IO mechanics inside a cell wall.. https://github.com/mayfield/cellulario And an example of using it to manage a multi-tiered scheme where a first layer of IO requests seeds another layer and the…

This looks really promising. I've often wanted to be able to do exactly this: run a bunch of async code in the middle of an otherwise synchronous block (classic example: writing a Django view which fires off a bunch of parallel HTTP API requests and continues once all of them have either returned or timed out).

Re: Making 1M requests with Python-aiohttp

#36
post #27

Earlier quoted context omitted.

I don't care for PHP as much as the next guy, but it's usually in the top 25 of the web framework benchmark (most of the other top langs are Java, Go and C++): https://www.techempower.com/benchmarks/

Just curious, what's up with this Ur language at both position 1 and 4? Never heard of it, and probably not experienced enough to make sense of it, but how is it that a language that doesn't even have a full official tutorial to its name beat out java, C++ and Go in those rankings by a factor of >2? I'm genuinely curious.

Ur (Ur/Web?) seems to be built very specifically for the exact things that this benchmark checks (dynamic web pages with SQL queries). So it's not surprising that those code paths are highly optimized in the language.

Re: Making 1M requests with Python-aiohttp

#37
post #35
post #16

I have a library for doing coordinated async IO in python that addresses some of the scheduling and resource contention issues hinted out in the later part of this post. It's called cellulario in reference to containing async IO mechanics inside a cell wall.. https://github.com/mayfield/cellulario And an example of using it to manage a multi-tiered scheme where a first layer of IO requests seeds another layer and the…

This looks really promising. I've often wanted to be able to do exactly this: run a bunch of async code in the middle of an otherwise synchronous block (classic example: writing a Django view which fires off a bunch of parallel HTTP API requests and continues once all of them have either returned or timed out).

That's almost exactly the use case I began with. It unapologetically requires python 3.5+ but if you're already there I'd be happy to see and support some of your use cases. Hit me up on github if you want to try it and need some guidance (the docs are nonexistent).

Re: Making 1M requests with Python-aiohttp

#38

Earlier quoted context omitted.

i keep looking for a reason to switch to python 3 and cant find one. Plus if I want to use the cool stuff in Pypy.. then I better not ! overall - very less reason to consider Py3 at all. Performance would have been one - if there were a comparison between gevent and asyncio.

>i keep looking for a reason to switch to python 3 and cant find one. Unicode? Not having to deal with encoding all over the place has been well worth switch to Python 3. If performance is a a huge issue, I honestly don't know why you would stay on Python (regardless of version) I wouldn't want to switch back to Python 2.7 is I can avoid it. There's honestly no reason not to go with 3.4 or 3.5 at this point, unless y…

This is a superficially trivial bit of syntactic sugar, but an example of the way small tweaks can provide big impact. This:

    > do_something(*some_args, *some_more_args)
is rocking my world right at the moment. That's a massive time saving feature I've been waiting for and worth the price of a 3.5 upgrade.

Re: Making 1M requests with Python-aiohttp

#39
post #10

1,000,000 requests in 52 minutes is just 320 req/sec. Am I missing something? What's so amazing about this? I just deployed some production feed that serves at 1955 requests/second on a cheap VPS in freaking PHP, one of the slowest languages out there.

Why you say is not amazing? Honestly curious here :)

Because you can get 540 req/s on Raspberry Pi 2 with Elixir/Phoenix.

http://blog.onfido.com/using-cpus-elixir-on-raspberry-pi2/

Re: Making 1M requests with Python-aiohttp

#40
post #31

"Everyone knows that asynchronous code performs better when applied to network operations" Ummm that seems a bit far reaching.

It depends on what "network operations" you are trying to do. For high-concurrency purposes, asynchronous programming is far more scalable (see: epoll/kqueue + state machines). For high-throughput, low-concurrency operations, it doesn't matter as much.

I happen to know of a very major tech company who scale is insane yet their core c++ code is based on highly tuned blocking threads. It's not a given that async is the only way to scale.
Post reply on HN