51–60 of 84 posts
Earlier quoted context omitted.
> Am I missing something? What's so amazing about this? The article is not about testing performance of a web server, but showcasing performance differences between synchronous and asynchronous code using asyncio. So, not about serving requests, but consuming.
Then he should change the title.
Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?
The 1 million in the title is misleading (1M per hour is nothing to write home about, only 278/sec). There are frameworks that are able hit 1M per minute plus (16,666/sec).
Here, mioco handling 10M http request per second(1) on my desktop:
https://github.com/dpc/mioco/blob/master/BENCHMARKS.md
1) with a bit of cheating http server.
With actual proper http parsing it goes down to 368K req/s, but that's still a lot.
Re the EADDRNOTAVAIL from socket.connect(), If you're connecting to 127.0.0.1:8080, then each connection from 127.0.0.1 is going to be assigned an ephemeral TCP source port. There are only a finite number of such ports available, on the order of ~30-50k, which limits the number of connections from a single address to a specific endpoint. If you're doing 100k TCP connections with 1k concurrent conections, it's feasibl…
Hacky way to get around that is to enable tcp_tw_reuse which will let you reuse ports, but it can be risky if you get a SYN from the previous connection that happens to lineup with segment number of the current connection (which will close your connection). Shouldn't happen often, and if you can tolerate a small amount of failure is an easy way to get around this limit.
[0] http://blog.davidvassallo.me/2010/07/13/time_wait-and-port-r...
Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?
I think async and aiohttp are game changes for Python 3.5. After working with Twisted callbacks for over a decade, it's a pleasure to write async code that does not use the callback approach (granted Twisted is a mature environment with lots to offer). I've switched to Python3.5 and aiohttp for all new web service applications. The coding style is clean, enjoyable to write, and easy to debug. Plus, I've never once be…
I have a library for doing coordinated async IO in python that addresses some of the scheduling and resource contention issues hinted out in the later part of this post. It's called cellulario in reference to containing async IO mechanics inside a cell wall.. https://github.com/mayfield/cellulario And an example of using it to manage a multi-tiered scheme where a first layer of IO requests seeds another layer and the…
This looks really promising. I've often wanted to be able to do exactly this: run a bunch of async code in the middle of an otherwise synchronous block (classic example: writing a Django view which fires off a bunch of parallel HTTP API requests and continues once all of them have either returned or timed out).
Earlier quoted context omitted.
Why you say is not amazing? Honestly curious here :)
Because it's trivial. I would be interested in anything doing 10,000+ req/sec on a cheap VPS. 320 is nothing. People achieve 2 million requests/second with C++ on EC2: https://medium.com/swlh/starting-a-tech-startup-with-c-6b5d5...
Well, for this particular start-up, I think c++ was an excellent choice (especially as they already knew c++!) -- but what if you could still run the service on a single service with python? You would still need 2-3 servers with the c++ version (failover, test etc).
Or it might turn out that for production load, you'd need 10 python server-instances. Sure, one server could handle this with c++ -- but you're not actually saving 39/40, you're only saving 9/10, because you didn't need "the full 40".
Not to mention, that it seems unlikely that the http request stuff is the limiting factor in a distributed OLAP system. So you might do 100 "OLAPS/s", and have them easily served by a 40Kreq/s python service.
"I guess we could have written it in Python to start off with but, economically, it would be a wastage of labor cost and time because, at some stage, we would have to scrap it for a C++ version to get the performance we need. The Python code will have no economic value once scrapped."
As mentioned, I think c++ was an excellent choice, but the above assumes they'd have to rewrite the entire system in c++, and scrap all the python. Granted, if you don't know python, but do know c++ very well -- it's doubtful that it would be faster to prototype in python than to just use c++.
But if (wild guesstimate) you could get 90% of the features, at 10% the loc/dependencies in python -- who's to say that wouldn't make sense? Maybe parts could be done in c/c++, or the project could move to pypy for enough of a speedup that no complete rewrite would be needed... etc.
Overall though, I really think this is a great illustration that high-performance compiled languages offer great performance, and can be rather pleasant to work with.
Earlier quoted context omitted.
I see you've been downvoted. Interesting, you said that you haven't found a reason to switch and someone looked at it and thought "How dare you not find a reason to switch, here let's teach you a lesson". But in large I agree. 3 hasn't provided enought of a carrot and 2 hasn't been enough of a pain for many people to want to switch. Especially when it comes to existing stable code bases. For new development, yes, man…
but Python 3 is not really an upgrade is it ? it is a very different language and most people who are pushing (downvoting?) for Python 3 dont seem to understand that. I have zero problems with Python 3 per se - but when the vast majority of the ecosystem is on Py2 and there is no difference in performance... then I see no reason to consider any breakages.
As a result, guys like you who are thinking rationally become the problem for being 'lazy' (acting in your own best interests which is exactly what everyone is doing) and are the enemy. People, especially newcomers, get tired of waiting for Python2 to 'die' and instead of put the onus on those who made the mistakes with Python3 (they've stuck it out, refusing to correct their own mistakes because that's more work), it becomes twisted and you are now the problem in their mind. Even though you were probably a part of what made Python successful to begin with. Amazing how that pans out, right?
Usually these illusions go away once a full time job is found, there are some but the vast majority are companies with big Python2 codebases that have features to deliver. If they move services anywhere from CPython it's to PyPy for the performance gain.
Earlier quoted context omitted.
i keep looking for a reason to switch to python 3 and cant find one. Plus if I want to use the cool stuff in Pypy.. then I better not ! overall - very less reason to consider Py3 at all. Performance would have been one - if there were a comparison between gevent and asyncio.
>i keep looking for a reason to switch to python 3 and cant find one. Unicode? Not having to deal with encoding all over the place has been well worth switch to Python 3. If performance is a a huge issue, I honestly don't know why you would stay on Python (regardless of version) I wouldn't want to switch back to Python 2.7 is I can avoid it. There's honestly no reason not to go with 3.4 or 3.5 at this point, unless y…
I see this a lot. That one doesn't care of performance that much to switch to, say, C++ or Go, doesn't mean one is OK with regressions to what they currently use.
Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?