Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reining in the thundering herd: Getting to 80% CPU utilization with Django

blog.clubhouse.com

Post reply on HN
Print view View original

61–70 of 139 posts

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#61

Post by mst »

ksec wrote: post #52
stu2010 wrote: post #36

Interesting to see this. It sounds like they're not on AWS, given that they mentioned that having 1000 instances for their production environment made them one of the bigger deployments on their hosting provider. If not for the troubles they experienced with their hosting provider and managing deployments / cutting over traffic, it possibly could have been the cheaper option to just keep horizontally scaling vs putti…

I was wondering that too, but there aren't that many common cloud provider that has 96 vCPU offering. I am also wondering on 144 Workers, on 96 vCPU which is not 96 CPU Core but 96 CPU thread . So effectively 144 Workers on 48 CPU Core possibly running at sub 3Ghz Clock Speed. But it seems they got it to work out in the end. ( May be at the expense of latency )

Assuming you're running a system where normal request/response handling blocks on database queries it's often optimal to have more workers than available cpu threads and 1.5x is a common rule of thumb to try first.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#62

Post by jimsimmons »

taylorhughes wrote: post #55
lmilcin wrote: post #44

Earlier quoted context omitted.

I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compare…

(CH employee here) The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that…

Don’t sweat it buddy. People here just want to stand on your toes and feel taller. Classic HN.

Velocity of development is priority #1 and having something that needs to be scaled is a monumental achievement.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#63

Post by mst »

lmilcin wrote: post #51
nomdep wrote: post #46

Earlier quoted context omitted.

So do you think using Django is stupid? I guess you think the same about every product that uses Ruby on Rails?

No, Django is not stupid. It is the decision to choose it to run load that will require 1000s of servers when it could be handled with 5-10 servers in another technology without more development effort.

I doubt they expected that level of request load that early on - I imagine the technology choice was made significantly before the whole pandemic thing started.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#64

Post by mst »

taylorhughes wrote: post #55
lmilcin wrote: post #44

Earlier quoted context omitted.

I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compare…

(CH employee here) The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that…

Plus, if he could've predicted the pandemic that far in advance there would probably have been plenty of not clubhouse ways to monetise that prescience ;)

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#65

Post by mst »

TekMol wrote: post #15
domino wrote: post #11

Earlier quoted context omitted.

Clubhouse is using CPython

Interesting. Is there a reason for this?

It's the standard and best supported approach and the level of speedup you get from PyPy is significantly workload dependent.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#66

Post by jimsimmons »

lmilcin wrote: post #60
taylorhughes wrote: post #55

Earlier quoted context omitted.

(CH employee here) The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that…

This is just silly excuse. The job of the cofounder is also to anticipate possible risks. And building your company on an astronomically inefficient technology sounds like a huge risk to me. Those 1000s of servers are probably a very significant cost with such small technical staff. Just by choosing the right technology for the problem, most of that cost could have been avoided. Django has nothing special in it that…

Python is not astronomically inefficient. Instagram serves like a billion users with it. Job of a cofounder is to build what people want. You can always scale in Silicon Valley by hiring people like you. You can’t build another viral app like clubhouse by hiring from the same crowd.

This may hurt you but the truth is scaling and software engineering is highly commoditised. That’s the whole point of being in the valley. You can hire people for such things and forget about it.

Clubhouse is not a tech company. They don’t have to care about being the best at infra

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#67

Post by mst »

JanMa wrote: post #45

Interesting to read that they are using Unix sockets to send traffic to their backend processes. I know that it's easily done when using HaProxy but I have never read about people using it. I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.

I do that every chance I can get.

At a guess, it's probably most loved by people picking old school simple architectures that aren't the sort of thing that goes viral.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#68

Post by ddorian43 »

latchkey wrote: post #38
lddemi wrote: post #20

Earlier quoted context omitted.

Author here. We do and did use autoscaling heavily but at a certain scale we just ran out of headroom on the smaller instance types we were using. Jumping to a much larger instance types meant that we will likely never run into those headroom issues again, plus solves other problems like faster spin up, better sidecar connection pooling and allows for a much higher hit rate on per instance caching.

You were autoscaling a single threaded process. You had 1000 connections coming in and scaling 1000 workers for those connections. Everything was filtered through gunicorn and nginx, which just adds additional latencies and complexity, for no real benefit. What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it.…

Now you just 5x their costs.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#69

Post by taylorhughes »

Ozzie_osman wrote: post #58
stingraycharles wrote: post #19

Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…

Yea you are right. It could be a service being down and requests piling up, or a cache key expiring and many processes trying to regenerate the value at the same time, etc. I think the article just used this phrase to describe something else. (Great article otherwise).

Phrase borrowed from excellent uWSGI docs https://uwsgi-docs.readthedocs.io/en/latest/articles/Seriali...

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#70

Post by fanf2 »

Ozzie_osman wrote: post #58
stingraycharles wrote: post #19

Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…

Yea you are right. It could be a service being down and requests piling up, or a cache key expiring and many processes trying to regenerate the value at the same time, etc. I think the article just used this phrase to describe something else. (Great article otherwise).

There is an explanation of this kind of thundering herd about 3/4 down this article https://httpd.apache.org/docs/trunk/misc/perf-scaling.html

The short version is that when you have multiple processes waiting on listening sockets and a connection arrives, they all get woken up and scheduled to run, but only one will pick up the connection, and the rest have to go back to sleep. These futile wakeups can be a huge waste of CPU, so on systems without accept() scalability fixes, or with more tricky server configurations, the web server puts a lock around accept() to ensure only one process is woken up at a time.

The term (and the fix) dates back to the performance improvement work on Apache 1.3 in the mid-1990s.

Post reply on HN