Hey - just wanted to give props to the author. I always enjoy Encore content when it pops up here.
Brand is cool too
The Thundering Herd Problem
11–20 of 38 posts
Re: The Thundering Herd Problem
#12Once had to interface with an external API for many of our client requests (booking requests for transportation). Due to certain combinations of queries, it could sometimes be slow to get a response if the requested journey would contain multiple legs and combinations. So we needed to have a high timeout for our requests to that system. Which was fine most of the time.
Unfortunately, if that system started to struggle, all our calls would be slow, or even time out. And of course us continuously spamming them wouldn't help (we were like 95% of their traffic). But that didn't just wreak havoc on their side, but it completely exhausted all our incoming connections, thread pools etc. just waiting for responses that would never come, with long timeouts.
A circuit breaker here was golden. When too many requests in a row were slow, we would break and stop requests from our side. That would allow them to recover on their side, and things on our side to fail fast and let other services still work.
A key here, was that the circuit breaker would allow a small percentage through still. How else would you detect things are back up again? And then slowly let more and more through if things are going fine (so not just open completely all at once).
Re: The Thundering Herd Problem
#13Hey - just wanted to give props to the author. I always enjoy Encore content when it pops up here. Brand is cool too
Although I just checked your pricing on a whim and it's the most expensive per member service I've ever seen for devs. By a LOT.
I mean I still like the brand but wow.
Re: The Thundering Herd Problem
#14Adding a little bit of random delay to any scheduled / triggered actions is always a good idea, and is usually very cheap to implement. If you're on OpenBSD, their cron supports randomized ranges using the ~ operator (see https://man.openbsd.org/crontab.5 >). Otherwise, you can use something like 'sleep $(($RANDOM % 60)) && some-task', but beware that $RANDOM has a range of 0-65535; you won't get the full, uniform 86…
For those who don't have the weird systemd allergy, systemd has many great features, including random scheduling, see `RandomizedDelaySec`.
Oh yes, systemd in general and timers in particular do have a whole bunch of awesome stuff - I just tend to forget any of it exists, it's my coping mechanism for dealing with "heterogeneous" environments. I still have CentOS6 machines in prod...
https://www.freedesktop.org/software/systemd/man/latest/syst...
Re: The Thundering Herd Problem
#15Props to encore to writing interesting blogs, also I just want to say I tried the encore product and it's awesome(I only used it for a hobby project but still)
Re: The Thundering Herd Problem
#16> user journeys
meaning, they tested that if the user does everything the way they're "supposed to" it works.
That explains a lot.
Re: The Thundering Herd Problem
#17As someone who has worked on very large networks for 15 years, the author left out some important bits about using caching with thundering herd problems. You really need to make sure that your cache settings will not generate a ton of requests to the origin when new content is requested. If you have a bunch of clients that are going to request content at nearly the exact same time (this happens a lot with live video…
Caching is generally way harder to get right than a lot of people probably think. Having delved into building caching systems that get the behavior and needs of the system just right, it took way more effort and research than I thought it would. You start with something simple and by the time you have chased down all the edge cases and tuned the cache behavior to your situation it is often not so simple after all.
Re: The Thundering Herd Problem
#18Once again, what TFA describes is NOT the thundering herd problem.
https://en.wikipedia.org/wiki/Thundering_herd_problem
> In computer science, the thundering herd problem occurs when a large number of processes or threads waiting for an event are awoken when that event occurs, but only one process is able to handle the event.
(emphasis mine)
Re: The Thundering Herd Problem
#19Earlier quoted context omitted.
Ironically, stream processing, which we tend to call "push" architecture, is largely the same solution.
Do you mean something like "at most once" delivery over UDP?
No, that's more like traditional load shedding. I'm talking more like Kafka/NATS/0mq/etc.
Re: The Thundering Herd Problem
#20Once again, what TFA describes is NOT the thundering herd problem. https://en.wikipedia.org/wiki/Thundering_herd_problem > In computer science, the thundering herd problem occurs when a large number of processes or threads waiting for an event are awoken when that event occurs, but only one process is able to handle the event . (emphasis mine)
Thundering herd has evolved beyond the original definition, that happens. Sometimes terms even flip their meaning (like how "one bad apple" has flipped it's meaning, or how "pre-optimization is the root of all evil" as used these days isn't what the original author actually meant, but in this case I think most of the new usages fit the basic idea, which is a large amount of unusual traffic happens outside of normal usage patterns.