I've always used nginx for my servers. Is HAProxy that much better to consider learning/switching?
Reining in the thundering herd: Getting to 80% CPU utilization with Django
91–100 of 139 posts
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#92Performance is the only thing that is holding me back to consider Python for bigger web applications. Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake. Pypy could be a solution as it seems to be about 6x faster on average. Is anybody here using Pypy for…
I feel like people let go of perf too easily.
When using something like Golang, I have apps doing normal CRUD-ish queries at 10k QPS, on 32c/64g machines. For most web apps, 10k QPS is much more than they will ever see, and the fact that it is all done in a single process means you could do really cool things with in-memory datastructures.
Instead, every single web app is written as a distributed system, when almost none of them need to be, if they were written on a platform that didn't eat all of their resources.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#93This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless no…
Could the discrepancy be explained by the type of responses? Sounds like an app like clubhouse might have lots of small, fast responses (like direct messaging), where very little of the response time is spent in application code. Does your API happen to do a lot of CPU-intensive stuff in application code?
Our app is also a messaging app. So lots of small & fast responses.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#94Earlier quoted context omitted.
Not if you do it right. a) you get to fire the devops person, which saves $150k+ a year. b) you add appropriate caching layers in front of everything. c) you spend time adding features, which generate revenue. I've done all of this before at scale. This whole case study was written about work I did [1]. Two devs, 3 months to release, first year was $80m gross revenue on $500/month cloud bills. Infinite scalability, z…
> you get to fire the devops person, which saves $150k+ a year. You are deluded or extremely short-sighted if you believe you can actually fire the devops guy. From my experience, the more you stray away from the conventional "dedicated server" paradigm the more you need a devops guy and you are in a very precarious position if you do fire him and something goes wrong.
You don't hire the devops person until you've scaled to the point that you need one.
Additionally, your thought of having my company held hostage by a single devops person is terrifying. Now you need two of them, which is even more expensive.
It is a great way to bootstrap a company by saving on a salary (or two) that can honestly be engineered out for a lot of SASS businesses. It worked super well for us... and calling someone who did $80m in the first year deluded seems well, rude.
But, if you start off designing systems that scale on their own, you are much better prepared for when you do get some fast growth than dealing with hiring a good devops person (which is extremely hard, as they say.. all the good ones are taken).
At the end of the day, the actual elephant in the room is that django was the wrong choice. You end up having to go through a lot of contortions to make things work, as evidenced by the blog post. The architecture doesn't make things easy to spin up quickly... which creates a lot of bottlenecks. There are better cloud-based solutions.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#95Interesting to read that they are using Unix sockets to send traffic to their backend processes. I know that it's easily done when using HaProxy but I have never read about people using it. I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.
It's standard way to connect things in UNIX and provides better performance. For example postgresql tcp+ssl is 175% slower than socket https://momjian.us/main/blogs/pgblog/2012.html#June_6_2012
But domain sockets only work between processes on the same machine, why would SSL be used in that case?
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#96Unfortunately HAProxy doesn't buffer requests*, which is necessary for a production deployment of gunicorn. And for anybody using AWS, ALB doesn't buffer requests either. Because of this I'm actually running both HAProxy and nginx in front of my gunicorn instances—nginx in front for request buffering and HAProxy behind that for queuing. If anybody is interested, I've packaged both as Docker containers: HAProxy queuin…
Couldn't Apache httpd just do all of that for you? mod_buffer provides request buffering, and mod_proxy_balancer provides load balancing capabilities.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#97Earlier quoted context omitted.
Phrase borrowed from excellent uWSGI docs https://uwsgi-docs.readthedocs.io/en/latest/articles/Seriali...
Funny reading this comment after reading the article > So many options meant plenty of levers to twist around, but the lack of clear documentation meant that we were frequently left guessing the true intention of a given flag. And then reading your link, they complain >inside the docs< that the docs aren't complete. I have no idea what to believe anymore :D
The uWSGI docs also say, in the section called "uWSGI developers are fu*!ing cowards": "why --thunder-lock is not the default when multiprocess + multithread is requested? This is a good question with a simple answer: we are cowards who only care about money."
Strange read.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#98Earlier quoted context omitted.
containers would've solved it one process per container, easy peasy
you now containers are just processes, right? This is what they did, but because they didn't need to schedule other jobs on the same machine, kubernetes or even docker would be overkill. In this case, simple VM orchestration seems like a fine solution.
Indeed,
but you wouldn't be thinking about instance sizes,
how many processes per instance and
wondering if you're hitting kernel limits with all the issues coming up
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#99Unfortunately HAProxy doesn't buffer requests*, which is necessary for a production deployment of gunicorn. And for anybody using AWS, ALB doesn't buffer requests either. Because of this I'm actually running both HAProxy and nginx in front of my gunicorn instances—nginx in front for request buffering and HAProxy behind that for queuing. If anybody is interested, I've packaged both as Docker containers: HAProxy queuin…
Couldn't Apache httpd just do all of that for you? mod_buffer provides request buffering, and mod_proxy_balancer provides load balancing capabilities.
Can Apache do request queuing?
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#100This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless no…
glad you are seeing such awesome performance with gevent+envoy! which part of our experience do you think is suspect?
So, in guincorn default mode (sync), the mode I'm assuming you're using. This means you really have 1 process handling 1 request at a time. The "thundering herd" problem really only applies to connection acceptance. Which is to say, that in the process of accepting a connection, it is possible to wake all idle processes that are waiting for a connection comes in (they will wake and hit EAGAIN and then go back to waiting.) Busy processes that are servicing requests (not waiting on the accept call) will not be woken, since they aren't waiting on a new request to come in. The "thundering herd" problem as I understand it, can indeed waste CPU cycles, but only on processes that aren't doing much anyways. I do however believe that `accept()` calls have been synchronized between processes on Linux for a while now to prevent spurious wakeups. You should verify you're actually doing spurious wakeups by using `strace` and seeing if you are seeing a bunch of `accept()` calls returning EAGAIN.
In gunicorn, `sync` mode does exhibit a rather pathological connection churn, because it does not support keep-alive. Generally, most load balancing layers already will do connection pooling to the upstream, meaning, your gunicorn processes won't really be accepting much connections after they've "warmed up". This doesn't apply in sync mode unfortunately :(. Connection churn can waste CPU.
Another thing to also note is that if you have 150 worker processes, but your load balancer only allows 50 connections per upstream, chances are 100 of your processes will be sitting there idle.
Something just doesn't feel quite right here.
EDIT: I do see mention of `gthread` worker - so you might be already able to support http-keepalives. If this is the case, then you should really have no big thundering herd problem after the LB establishes connections to all the workers.