Kinda funny they decided paying a ton of money to aws was ok but paying for nginx plus was not
ClubHouse runs on AWS?
Reining in the thundering herd: Getting to 80% CPU utilization with Django
41–50 of 139 posts
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#42Kinda funny they decided paying a ton of money to aws was ok but paying for nginx plus was not
ClubHouse runs on AWS?
Hm actually might be google based on what their traffic is going to (i only looked just now). Ok now it makes more sense why support wasn’t able to figure this out =)
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#43If you’re delegating your load balancing to something else further up the stack and would prefer a simpler WSGI server than Gunicorn, Waitress is worth a look: https://github.com/pylons/waitress
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#441M requests per minute on 1000 web instances is not an achievement, it is a disaster. It is ridiculous people brag about it. Guys, if you have budget maybe I can help you up this by couple orders of magnitude.
To be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewriting the whole thing. It seemed to be all about how to extract the most performance from the lemon they had to deal with. I found the linked reference really informative too: https://rachelbythebay.com/w/2020/03/07/costly/
I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks.
Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compared to their 1M per 1000 nodes.
This is what I am typically getting from a simple setup with Java, WebFlux and MongoDB with a little bit of experience on what stupid things not to do but without spending much time fine tuning anything.
I think bragging about performance improvements when your design and architecture is already completely broken is at the very least embarrassing.
> poor hindsight from original developer (co-founder)
Well, you have a choice of technologies to write your application in, why chose one that sucks so much when there are so many others that suck less?
It is not poor choice, it is lack of competency.
You are co-founder and want your product to succeed? Don't do stupid shit like choosing stack that already makes reaching your goal very hard.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#45Interesting to read that they are using Unix sockets to send traffic to their backend processes. I know that it's easily done when using HaProxy but I have never read about people using it.
I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#46Earlier quoted context omitted.
To be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewriting the whole thing. It seemed to be all about how to extract the most performance from the lemon they had to deal with. I found the linked reference really informative too: https://rachelbythebay.com/w/2020/03/07/costly/
I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compare…
So do you think using Django is stupid? I guess you think the same about every product that uses Ruby on Rails?
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#47Earlier quoted context omitted.
By this logic, why not Java, C++, Rust, Go, C#? They’re all web-capable and blow the doors off PHP, Python, etc.
C#, Java, C++ need application servers, no? "Serverless" scales infinitely due to its simpler request/response lifecycle.
Serverless is too overloaded a term to have any meaning. I'm not really seeing how Python or PHP "scales infinitely" in any way that C#, Java, C++ couldn't.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#48Earlier quoted context omitted.
By this logic, why not Java, C++, Rust, Go, C#? They’re all web-capable and blow the doors off PHP, Python, etc.
Yes all of those would be way better options than Python and probably PHP. Well maybe not C++. You'd have to be pretty crazy to have web developers writing security sensitive code in C++. The "blame our co-founder for the choice" bit is exactly what that graph about the cost of defects vs how early they are fixed is talking about. If they had just picked Go or Java right at the start they wouldn't have had to expend…
This thread arose from a person that said "Use PHP", as an argument to using Python. It's a silly argument.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#49Kinda funny they decided paying a ton of money to aws was ok but paying for nginx plus was not
I kinda get that honestly. It’s why I’ll spend $20 without even thinking for take out but not spend $2 for an app. It’s because the cost off the software is way way more than the money. It’s a commitment to actually use it and integrate it, deal with their sales team, talk to purchasing, handle licensing, and introducing friction to replacing it or using tools that don’t integrate well because “well we already pay fo…
The buy in into AWS is much, much larger then using a piece of software though.
Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django
#50HAProxy is a beautiful tool but it doesn't buffer requests that is why NGINX is recommended in front of gunicorn otherwise it's suspectible to slowloris attack. So either cloubhouse can be easily DDOS'd right now or they have some tricky setup that prevents slow post reqests reaching gunicorn. In the blog post they don't mention that problem while recommend others to try and replace NGINX with HAPRoxy.