Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reining in the thundering herd: Getting to 80% CPU utilization with Django

blog.clubhouse.com

Post reply on HN
Print view View original

81–90 of 139 posts

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#81

Post by lmilcin »

yuliyp wrote: post #75
lmilcin wrote: post #32

1M requests per minute on 1000 web instances is not an achievement, it is a disaster. It is ridiculous people brag about it. Guys, if you have budget maybe I can help you up this by couple orders of magnitude.

Knowing nothing else, it's hard to know if this is good or not. It's 16 requests per second. Are those requests something like "Render a support article" or are they "Give the user a ranked feed of what they should see on their home screen"? Is most of the logic run by the web server or some combination of app servers / backend services behind it? What kind of hardware does the web server have? All of those would aff…

Well, to give you an idea, I am working on an service that implements rather complex business process involving fetching data from multiple sources, parsing binary blobs with market data in proprietary format, saving results to a database and so on. And it does around 10k requests per second on a single relatively normal node (8 cores, 64GB ram, etc.)

And no, it does not require any special tricks. It is regular Java / WebFlux / REST / MongoDB backend service.

CPUs can do really a lot and if your node processes 16 requests per second on a multi-core machine then you are using billions of clock cycles and gigabytes of possible transfer to memory for a single request. Something is not quite right...

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#82

Post by lmilcin »

jimsimmons wrote: post #66
lmilcin wrote: post #60

Earlier quoted context omitted.

This is just silly excuse. The job of the cofounder is also to anticipate possible risks. And building your company on an astronomically inefficient technology sounds like a huge risk to me. Those 1000s of servers are probably a very significant cost with such small technical staff. Just by choosing the right technology for the problem, most of that cost could have been avoided. Django has nothing special in it that…

Python is not astronomically inefficient. Instagram serves like a billion users with it. Job of a cofounder is to build what people want. You can always scale in Silicon Valley by hiring people like you. You can’t build another viral app like clubhouse by hiring from the same crowd. This may hurt you but the truth is scaling and software engineering is highly commoditised. That’s the whole point of being in the valle…

> Python is not astronomically inefficient.

Well, it is. It is a fact.

https://rachelbythebay.com/w/2020/03/07/costly/

> Clubhouse is not a tech company.

When you spin 1000s of nodes you need some tech competency.

Or in other words, if it blew one day and there would be a link to writeup on HN, people would be asking "They had 1000s of servers and nobody competent to maintain it?"

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#83

Post by aeyes »

latchkey wrote: post #38
lddemi wrote: post #20

Earlier quoted context omitted.

Author here. We do and did use autoscaling heavily but at a certain scale we just ran out of headroom on the smaller instance types we were using. Jumping to a much larger instance types meant that we will likely never run into those headroom issues again, plus solves other problems like faster spin up, better sidecar connection pooling and allows for a much higher hit rate on per instance caching.

You were autoscaling a single threaded process. You had 1000 connections coming in and scaling 1000 workers for those connections. Everything was filtered through gunicorn and nginx, which just adds additional latencies and complexity, for no real benefit. What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it.…

According to the article they have a monolithic Django application so this will have at least a couple of seconds start-up time. That is not a good match for Cloud Functions.

Django also has in-memory caches, for example for templates which can be extremely slow (seconds) and CPU intensive to render. So you really don't want to have AWS or Google restart your application on AppEngine whenever they feel like it.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#84

Post by mixmastamyk »

TekMol wrote: post #5

Performance is the only thing that is holding me back to consider Python for bigger web applications. Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake. Pypy could be a solution as it seems to be about 6x faster on average. Is anybody here using Pypy for…

Backend controller performance is rarely a bottleneck, and if raw-compute still is there are a number of ways to speed it up, such as cython and/or work queues.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#85

Post by jhgg »

This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless now that we're using envoy, so that'll probably go away soon.

Works amazingly well! We run our python API tier at 80% target CPU utilization.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#86

Post by swiley »

sdze wrote: post #14
void_mint wrote: post #9

Earlier quoted context omitted.

By this logic, why not Java, C++, Rust, Go, C#? They’re all web-capable and blow the doors off PHP, Python, etc.

C#, Java, C++ need application servers, no? "Serverless" scales infinitely due to its simpler request/response lifecycle.

Php needs an application server as well.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#87

Post by scrollaway »

aeyes wrote: post #83
latchkey wrote: post #38

Earlier quoted context omitted.

You were autoscaling a single threaded process. You had 1000 connections coming in and scaling 1000 workers for those connections. Everything was filtered through gunicorn and nginx, which just adds additional latencies and complexity, for no real benefit. What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it.…

According to the article they have a monolithic Django application so this will have at least a couple of seconds start-up time. That is not a good match for Cloud Functions. Django also has in-memory caches, for example for templates which can be extremely slow (seconds) and CPU intensive to render. So you really don't want to have AWS or Google restart your application on AppEngine whenever they feel like it.

There's a few reasons why this scenario wouldn't be a good fit for cloud functions, but that "couple of seconds start-up time" can be almost entirely removed from the equation by keeping the Django instance alive (all cloud function type offerings will have a concept of cold and warm starts, and some way to control persistence across calls on the same "instance").

I've run Django on AWS Lambda in a a scenario that scaled between 25-250 calls per second depending on time of day (for a runtime of 5-30 sec). Moving Django's bootstrapping so it would stay warm across calls was very easy.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#88

Post by void_mint »

sdze wrote: post #54
void_mint wrote: post #48

Earlier quoted context omitted.

This thread arose from a person that said "Use PHP", as an argument to using Python. It's a silly argument.

It was just a silly remark about the snail-like performance of Python. Another silly thing: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

I'm not really seeing what's silly about what you're linking. I'm seeing Java blowing the doors off of PHP, which matches my previous assertion.

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#89

Post by lddemi »

jhgg wrote: post #85

This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless no…

glad you are seeing such awesome performance with gevent+envoy! which part of our experience do you think is suspect?

Re: Reining in the thundering herd: Getting to 80% CPU utilization with Django

#90

Post by mac-chaffee »

jhgg wrote: post #85

This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless no…

Could the discrepancy be explained by the type of responses?

Sounds like an app like clubhouse might have lots of small, fast responses (like direct messaging), where very little of the response time is spent in application code. Does your API happen to do a lot of CPU-intensive stuff in application code?

Post reply on HN