Live data from Hacker News

The August 17 outage

github.blog

531–540 of 804 posts

Re: The August 17 outage

#531
> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.

This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".

When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.

Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.

Re: The August 17 outage

#532
post #486

Earlier quoted context omitted.

Are they happy about the circumstances of AI too? Higher resource consumption and the set back in CO2 reduction?

you seem to be uninformed. This should help: https://www.youtube.com/watch?v=H_c6MWk7PQc

> raised heat is a kind of pollution

> water is not unlimited

> you don't want to release a bunch of acid into a river

> I know next to nothing about this

> I know basically nothing

> corn is one of the thirstiest major crops grown in the US

What is this rant supposed to inform? Whats wrong with OP being concerned about the costs of operating a DC?

Re: The August 17 outage

#533
post #93

Earlier quoted context omitted.

Don’t forget jitter!

I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference. Do you know of a documented instance w…

Even if you're operating at a scale where thundering herd won't crash your service, introducing exponential backoff + jitter to your retries also helps when you're just viewing the logs. Massively reduces log spam when you make changes.

Re: The August 17 outage

#534
post #44

Earlier quoted context omitted.

Exponential backoff is your friend... too few people use it.

Exponential is also overkill (even with jitter as others have mentioned). I seem to remember there was a "you failed 5 PIN entries in a row, please wait 500000 seconds before you retry" on Apple phones. So, you probably also want a sensible max... which makes exponential a bit pointless. Just do a basic fixed delay + (large, e.g. 0.5 x the delay) jitter and you'll be fine for most things. You can add a bit of cumulat…

Cap the number of retries before the user has to manually try again. That's fine.

Re: The August 17 outage

#535

Earlier quoted context omitted.

None of the things you mention - indexing or secret scan would be done individually for each commit. As I already said, this would be a log of all commits in the range pushed - it would be scanned once for those things. There is no need for a loop running over a range of commits and processing each one.

There 100% is at least for things like secret scanning and message parsing. Secret scanning needs to make sure my repo as a whole has no secrets. It’s not acceptable to have 1 commit introducing it and 1 removing it because the secret is still recoverable. Every commit is also surely an entry in a database somewhere. I can navigate in GitHub directly to any individual commit so there is definitely some overhead of so…

You are right about secret scanning, but its worth noting that is only enabled by default on public repositories. It is an extra paid feature for organizations on a teams or enterprise subscription, and isn't available at all for individual owned repositories outside of an enterprise subscription.

And yes, I agree there is indexing of commits, but that is a batch insert from a log.

Re: The August 17 outage

#536
post #486

Earlier quoted context omitted.

Are they happy about the circumstances of AI too? Higher resource consumption and the set back in CO2 reduction?

you seem to be uninformed. This should help: https://www.youtube.com/watch?v=H_c6MWk7PQc

What make you think I‘m talking about water consumption?

The construction of data centers needs resources and also creates more the CO2.

The energy for these data centers is often created through fossil fuels which also creates additional CO2

What do you think why Google and MS scrapped their CO2 reduction targets

Re: The August 17 outage

#538

Earlier quoted context omitted.

> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour That wouldn't be a unit test - that's more like an end-to-end or integration test. Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.

My employer has 0% test coverage lol. I've begged and pleaded, but the claim is that "risk is low" and "that's what QA is for." Hell, I've complained to senior management about how there are senior devs that forgo backend validation. It's truly Hell in the trenches sometimes. Some days, I would seriously rather work at Wendy's.

> Some days, I would seriously rather work at Wendy's.

Narrator: He would not.

Re: The August 17 outage

#539

Distributing across different services wouldn't be a bad idea.... I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but... Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (exce…

The point is, GitHub is not free for me (we pay) yet we were also down?

IMO definiely time to segregate the free and the paid service. Otherwise those of us who need the service to stay up will have to look for alternatives.

The post mortem conclusion should've been to put usage limits on the free tier so paid users aren't killed by them.

Re: The August 17 outage

#540
I have setup a gitea instance on my gitea server which I think is good for me and GitHub both. For one, I dont have to worry about GitHub service outages, and GitHub gets to be free from my toy (and mostly AI slop) projects that no one else will ever read/use/participate in :-)
Post reply on HN