Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reining in the thundering herd: Getting to 80% CPU utilization with Django

blog.clubhouse.com

Post reply on HN
Print view View original

51–60 of 139 posts

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#51

Post by lmilcin »

nomdep wrote: post #46
lmilcin wrote: post #44

Earlier quoted context omitted.

I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compare…

So do you think using Django is stupid? I guess you think the same about every product that uses Ruby on Rails?

No, Django is not stupid.

It is the decision to choose it to run load that will require 1000s of servers when it could be handled with 5-10 servers in another technology without more development effort.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#52

Post by ksec »

stu2010 wrote: post #36

Interesting to see this. It sounds like they're not on AWS, given that they mentioned that having 1000 instances for their production environment made them one of the bigger deployments on their hosting provider. If not for the troubles they experienced with their hosting provider and managing deployments / cutting over traffic, it possibly could have been the cheaper option to just keep horizontally scaling vs putti…

I was wondering that too, but there aren't that many common cloud provider that has 96 vCPU offering.

I am also wondering on 144 Workers, on 96 vCPU which is not 96 CPU Core but 96 CPU thread. So effectively 144 Workers on 48 CPU Core possibly running at sub 3Ghz Clock Speed. But it seems they got it to work out in the end. ( May be at the expense of latency )

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#53

Post by lddemi »

kvalekseev wrote: post #50

HAProxy is a beautiful tool but it doesn't buffer requests that is why NGINX is recommended in front of gunicorn otherwise it's suspectible to slowloris attack. So either cloubhouse can be easily DDOS'd right now or they have some tricky setup that prevents slow post reqests reaching gunicorn. In the blog post they don't mention that problem while recommend others to try and replace NGINX with HAPRoxy.

1. HAProxy does support request buffering https://cbonte.github.io/haproxy-dconv/2.2/configuration.htm...

2. our load balancer buffers requests as well

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#54

Post by sdze »

void_mint wrote: post #48
IshKebab wrote: post #26

Earlier quoted context omitted.

Yes all of those would be way better options than Python and probably PHP. Well maybe not C++. You'd have to be pretty crazy to have web developers writing security sensitive code in C++. The "blame our co-founder for the choice" bit is exactly what that graph about the cost of defects vs how early they are fixed is talking about. If they had just picked Go or Java right at the start they wouldn't have had to expend…

This thread arose from a person that said "Use PHP", as an argument to using Python. It's a silly argument.

It was just a silly remark about the snail-like performance of Python.

Another silly thing:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#55

Post by taylorhughes »

lmilcin wrote: post #44
smashed wrote: post #39

Earlier quoted context omitted.

To be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewriting the whole thing. It seemed to be all about how to extract the most performance from the lemon they had to deal with. I found the linked reference really informative too: https://rachelbythebay.com/w/2020/03/07/costly/

I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compare…

(CH employee here)

The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that gives you the best shot at capturing the lightning. Django seemed to help!

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#56

Post by kvalekseev »

lddemi wrote: post #53
kvalekseev wrote: post #50

HAProxy is a beautiful tool but it doesn't buffer requests that is why NGINX is recommended in front of gunicorn otherwise it's suspectible to slowloris attack. So either cloubhouse can be easily DDOS'd right now or they have some tricky setup that prevents slow post reqests reaching gunicorn. In the blog post they don't mention that problem while recommend others to try and replace NGINX with HAPRoxy.

1. HAProxy does support request buffering https://cbonte.github.io/haproxy-dconv/2.2/configuration.htm... 2. our load balancer buffers requests as well

From HAProxy mailing list about http_buffer_request option https://www.mail-archive.com/haproxy@formilux.org/msg23074.h...

> In fact, with some app-servers (e.g. most Ruby/Rack servers, most Python servers, ...) the recommended setup is to put a fully buffering webserver in front. Due to it's design, HAProxy can not fill this role in all cases with arbitrarily large requests.

A year ago I was evaluating recent version of HAProxy as buffering web server and successfully run slowloris attack against it. Thus switching from NGINX is not a straightforward operation and your blog post should mention http-buffer-request option and slow client problem.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#57

Post by polote »

I wouldn't be very proud of writing an article like that.

Usually engineering blogs exists to show that there are fun stuff to do in a company. But here it just seems they have no idea, what they are doing. Which is fine, I'm classifying myself in the same category.

Reading the article I don't feel like they have solved their issue, they just created more future problems

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#58

Post by Ozzie_osman »

stingraycharles wrote: post #19

Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…

Yea you are right. It could be a service being down and requests piling up, or a cache key expiring and many processes trying to regenerate the value at the same time, etc.

I think the article just used this phrase to describe something else. (Great article otherwise).

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#59

Post by kvalekseev »

JanMa wrote: post #45

Interesting to read that they are using Unix sockets to send traffic to their backend processes. I know that it's easily done when using HaProxy but I have never read about people using it. I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.

It's standard way to connect things in UNIX and provides better performance. For example postgresql tcp+ssl is 175% slower than socket https://momjian.us/main/blogs/pgblog/2012.html#June_6_2012

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#60

Post by lmilcin »

taylorhughes wrote: post #55
lmilcin wrote: post #44

Earlier quoted context omitted.

I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compare…

(CH employee here) The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that…

This is just silly excuse.

The job of the cofounder is also to anticipate possible risks.

And building your company on an astronomically inefficient technology sounds like a huge risk to me.

Those 1000s of servers are probably a very significant cost with such small technical staff. Just by choosing the right technology for the problem, most of that cost could have been avoided.

Django has nothing special in it that would allow building applications faster than in a lot other frameworks that are also much more efficient.

So it is just a matter of simple choice.

Nobody expects people to write webapps in C++ or Rust. Just don't choose technology that is famous for being inefficient.

Post reply on HN