Earlier quoted context omitted.
When you have an outage, you should not retry at all. Exponential backoff is exactly how you get cascading outages. If service A fails a request to service B and decides to exponentially back off, now service A is holding open an end user request that will claim resources on service A. Fast forward ten minutes and the service B degradation has metastasized into a service A degradation. And even after service B has re…
> To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients. basically enumerate your errors, and depending on the type, retry or just return/forward that same "dont retry this" error?
The August 17 outage
401–410 of 804 posts
Re: The August 17 outage
#402Earlier quoted context omitted.
$1.2 billion of costs on revenue of $1-$2b certainly seems like a big concern.
They do not throw away the servers every year..
Re: The August 17 outage
#403Earlier quoted context omitted.
I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.
You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse.
The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated).
The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification.
I've seen cases which had:
1. Retries on the client side (browser or mobile app). 2. Retries on the BFF. 3. Retries on Microservice A used by the BFF. 4. Retries on Microservice B used by Microservice A. 5. Retries on Critical Service C used by Microservice B.
Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C.
We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day.
A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination.
[1] This was the CloudFlare issue mentioned in this thread https://blog.cloudflare.com/5-december-2025-outage/
Re: The August 17 outage
#404Earlier quoted context omitted.
AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.
I mean, isn't that the dream? I don't know if that's sarcasm or not. I know it doesn't work , but that's the future we've been promised, right?
Re: The August 17 outage
#405I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh No side projects? Nothing? Just seems odd.
Re: The August 17 outage
#406Earlier quoted context omitted.
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…
CAP theorem. Pick one of those.
Re: The August 17 outage
#407Earlier quoted context omitted.
Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.
Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering. Source: decades of operational pain.
Re: The August 17 outage
#408Earlier quoted context omitted.
When a service that was already the primary git hosting provider for most of the world for 20 years grows at that rate its not mundane and its not comparable to any example.
AWS saw this growth every year for two decades, hyper growth tech sees it all the time
That does not seem to be true - which two-decade period are you talking about? AWS has only been around for ~20 years, and I just reviewed a 10 year period, and not a single one of those years saw doubling in the whole year, let alone doubling in a few months. Which 20 year period are you referring to, and are you referring to doubling every few months over that 20 year period?
Re: The August 17 outage
#409Earlier quoted context omitted.
Your highly available system is probably somewhat important, otherwise you won’t have invested in making it HA. While your premise holds for happy cases, when you do have a cascading series of outages, not using exponential backoff is just adding a self-inflicted DoS to when you do go down. I don’t really follow your premise and can’t really articulate many cases for when you shouldn’t use exponential backoff. Maybe…
When you have an outage, you should not retry at all. Exponential backoff is exactly how you get cascading outages. If service A fails a request to service B and decides to exponentially back off, now service A is holding open an end user request that will claim resources on service A. Fast forward ten minutes and the service B degradation has metastasized into a service A degradation. And even after service B has re…
Even this is not enough, since you cannot always reliably know whether service B is dead or suffers an intermittent issue that can be safely retried just from looking at a single failure.
The classic solution, in the monolith/few-services world would be a circuit breaker. High failure rates on any service trigger a circuit breaker in the services calling it, and they'll wait for a cooldown period before trying again.
When you move to a massive microservice architecture with hundreds or thousands of microservices, setting up circuit breakers manually becomes very hard to track and do reliably. Service meshes like Istio make this slightly easier, but they still don't let you verify that all possible paths have circuit breakers and that retries are not excessive etc.
Re: The August 17 outage
#410Earlier quoted context omitted.
Strange to think they are probably triaged by LLMs at this point
AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.
https://github.com/oven-sh/bun/pull/39743 https://github.com/oven-sh/bun/pull/39735