I don't know where else to ask this but it's killing me. Does anyone know what the hell that GitHub physical CD thing was about? Did anyone in the world get theirs?
The August 17 outage
251–260 of 804 posts
Re: The August 17 outage
#252Earlier quoted context omitted.
Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering. Source: decades of operational pain.
This is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems. It's also true that we're witnessing unprecedented levels of scale.
I think the “thundering herd” problem is pretty extensively documented
Re: The August 17 outage
#253> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…
Maybe the retry logic was vibecoded instead of using an existing hardened library. After all, according to Twitter, nobody is looking at the code anymore.
> instead of using an existing hardened library
A lot of retry libraries I’ve seen require the user to configure them. You can use a library with all the right settings, but if you configure it wrong, you are really no better off than if you hadn’t
Re: The August 17 outage
#254> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…
> because the node you hit is rolling/broken/overloaded, but the others aren't
Retries in such a situation should be handled internally with the client at most responsible for failing over with a circuit breaker to another zone. Having the client auto retry right away is not something that behaves well as shown here, even if in the happy path it happens to stimulate increased availability without actually investing in the proper architecture for it
Re: The August 17 outage
#255Earlier quoted context omitted.
If you’re the size of GitHub and you’ve been running your infra for years with very little variation in traffic patterns you have a strong incentive to optimise costs for that existing behaviour.
I am afraid thats now how infrastructure works from what I have seen. The number that really matters is QPS. For any system the QPS varies through out the day and across the week and months. Most design considerations easily absorb any 2X increase. Pick up any company and the chances are that the servers are over provisioned, no one takes chances specially with critical components. What you have going on with Github…
How much infra have you seen of the top 100 sites in the world? I've worked on multiple top 10, and absorbing a 2x increase (and the peak is very likely more than 2x) is a very very hard problem that would cause hundreds of pagers to go off and load shedding to very high degrees. There is just not tons of unused capacity lying around in wait at the scale of github. "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.
Re: The August 17 outage
#256Earlier quoted context omitted.
Do you have some GitHub architectural knowledge you’d like to share with us? A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit. There’s a difference between pushing 1 commit and 100.
> There’s a difference between pushing 1 commit and 100. There isn’t much. GitHub doesn’t run actions separately for each commit. It runs them on pushes. I’m trying to think of a thing that would happen for each commit in each push and coming up blank. It does things like scan for references to issues to index, but it would just scan the log for a range. I did disagree with GP though because there is no reason to ass…
> It runs them on pushes
Sure, because pushes are how you update a reference. That’s really what triggers an action: a reference changing. And there could be a bunch of those in a push.
A commit costs storage, you’ve got secret scanning, it needs to be indexed in a way that can be referenced in commit messages and comments, a commit message itself can close issues or reference other PRs, stored and served individually and immediately via the web UI or git clients, etc etc.
It’s also like… the core unit of git.
Re: The August 17 outage
#257Earlier quoted context omitted.
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
Layoffs by the 10s of thousands, food prices out of control, people barely able to afford gas. At least some tech bros can launch their 50th B2B SaaS. I didn't realize a booming economy sucked so much.
The exciting thing about AI is precisely that it'll let software move beyond Yet Another B2B SaaS and into doing useful things in the real world. I regularly ride in driverless cars! That was the stuff of science fiction when I was a kid.
If you're worried about food prices, you should be happy that robots will make agriculture less labor-intensive and bring prices down.
Re: The August 17 outage
#258Earlier quoted context omitted.
The whole point of exponential backoff is that the first retry can be quick.
The right answer is for the RPC framework to accurately communicate "try again on another node" vs "don't try again, just hard fail". When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will tim…
This is why you have circuit breakers upstream. Not on every individual instance.
Re: The August 17 outage
#259Earlier quoted context omitted.
The absolute number doesn’t matter nearly as much as the change in rate of growth. The number of commits had not been doubling every six months at GitHub for a long time. At Amazon if traffic volume consistently doubled every six months that is actually quite a lot easier to plan for, it just becomes part of everything they do from very early on.
im not buying that github is a unique engineering problem, harder than the rest of hyperscalars.
No one said it is unique. But if you take an infrastructure and engineering org that had been growing at 10% a year for a decade, you are going to have a different set of capabilities and practices in place. Adapting to a new reality of doubling every few months will predictably produce failures anywhere. GitHub is not unique in that regard.
Re: The August 17 outage
#260"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.
It's almost double. If you have scaling prepared, it should be linear, but I doubt they've prepared for this.