101–110 of 139 posts
Earlier quoted context omitted.
Couldn't Apache httpd just do all of that for you? mod_buffer provides request buffering, and mod_proxy_balancer provides load balancing capabilities.
Can Apache do request queuing?
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#listen...
Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…
That... doesn't have much to do with the thundering herd problem. It also doesn't make much sense as a concept on its own merits -- say you come in to work and your inbox is full enough for three inboxes. Does that fact, in itself, mean that you decide you're done for the day? No, it just means you have a much longer queue to work through than usual.
The thundering herd problem refers to what happens when (1) a bunch of agents come to you for something while you're busy; (2) you tell them all "I'm busy, go away and come back later"; and (3) the come-back-later time you give to each of them is identical, so they all come back simultaneously.
And that's exactly what's happening here, except that instead of giving each worker thread a come-back-later time when it asks for work, you're receiving work, sending out individual messages to every worker saying "hey, I'm not busy anymore, come back RIGHT NOW and get some more work", and then rejecting all but one of the thundering herd that shows up. The reason the Gunicorn docs and the uWSGI docs both refer to this as a "thundering herd" problem is that it's a near-perfect match for the problem prototype. The only difference is that, instead of giving out identical come-back-later times to worker threads as they ask you for work, you tell them to wait for a notification that includes a come-back-later time, and then when you get one piece of work you fire off that notification separately to every sleeping thread, including identical come-back-later times in each one.
Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…
One solution is to use a soft starter which slow brings the motor up to speed.
Famous last words, but I get the sense that the need to handle this sort of load on Clubhouse is plateauing and will decline from here. The app seems to have shed all the people that drew other people initially and lost its small, intimate feel and has turned into either crowded rooms where no one can say anything, or hyper specific rooms where no one has anything to say. Good article though! I’ve dealt with these ex…
I have been through this journey, we eventually migrated to Golang and it saved a ton of money and firefighting time. Unfortunately, python community hasnt been able to remove GIL, it has its benefits (especially for single threaded programs), but I believe the cost (lack of concurrent abstractions. async/await doesn't cut it) far outweigh it.
Apart from what the article mentions, other low hanging fruits worth exploring are
[1] Moving under PyPy (this should give some perf for free)
[2] Bifurcate metadata and streaming if not already. All the django CRUD stuff could be one service, but the actual streaming should be separated to another service altogether.
Performance is the only thing that is holding me back to consider Python for bigger web applications. Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake. Pypy could be a solution as it seems to be about 6x faster on average. Is anybody here using Pypy for…
Typescript is a nicer language than Python in many ways and it doesn't suffer from Python's crippling performance issues or dubious static typing situation. Plus you can run it in a browser so there's only one language to learn.
Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…
Performance is the only thing that is holding me back to consider Python for bigger web applications. Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake. Pypy could be a solution as it seems to be about 6x faster on average. Is anybody here using Pypy for…
I feel like people let go of perf too easily. When using something like Golang, I have apps doing normal CRUD-ish queries at 10k QPS, on 32c/64g machines. For most web apps, 10k QPS is much more than they will ever see, and the fact that it is all done in a single process means you could do really cool things with in-memory datastructures. Instead, every single web app is written as a distributed system, when almost…
I've been told off in code review for using Python's concurrent.futures.ThreadPoolExecutor to run some http requests (making the code finish N times faster, in a context where latency mattered) "because it's hard to reason about".
Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…
Your problem is a real problem though. Where I worked, we would call that backlog, and we would manage it with 'floodgates' ... When the system is broken, close the gates, and you need to open them slowly.
In an ideal world, your system would self-regulate from dead to live, shedding load as necessary, but always making headway. But sometimes a little help is needed to avoid the feedback loop of timed out client requests that still get processed on the server keeping the server in overload.
Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the…
> but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled. That... doesn't have much to do with the thundering herd problem. It also doesn't make much sense as a concept on its own merits -- say you come in to work and…
If my SLA is 24 hour response time, and the inbox is FIFO, and I can't drop old messages, I'm most likely not hitting the SLA. If they all came in overnight, I'll hit the SLA for day 1, but I will be busy all of day 2 and 3 and never respond on time. If after day 1, I get a days worth of messages every day, I'll never catch up.