Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

641–650 of 804 posts

Re: The August 17 outage

#641

Post by afc »

vasco wrote: post #560
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle. Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.

Google's (or any similar large company's, really) efficiency requirements would never allow for such levels of waste. It would be incredibly irresponsible to allow just any random small server to unexpectedly use the entirety (or even just a non-trivial percentage) of a Google-scale datacenter's capacity. Do you evict other jobs? Do you run your datacenters cold to prepare for this eventuality? Both are terrible options.

Suddenly you push an update to your tiny server or one of its thousands of clients and, because of some bug or misconfiguration, traffic or per-request-resource-consumption grows 1000X or more. Given the size of the looong tail of small servers at Google, this would be a daily occurrence.

Re: The August 17 outage

#642

Post by ChrisMarshallNY »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

> this complex system collapses (rather than degrade gracefully) when demand exceeds capacity

This.

It's easy to be an "armchair quarterback," here, though.

Handling stuff like this, needs to be planned for, from the start. I suspect that a lot of the issues are because GitHub is something that started small (and probably quickly), and has accreted. Things like Facebook are in a similar boat.

Re: The August 17 outage

#643

Post by nickelpro »

tcmart14 wrote: post #419
nickelpro wrote: post #365

Earlier quoted context omitted.

Do you think the load is going away? The current infrastructure cannot handle the new load requirements. Either the infrastructure must change, or they must start denying users the ability to use the infrastructure.

I may have come out with a different interpretation than you did of GP's comment. I see how you got to yours. But the way I read them saying they should slow down was, maybe slow down on new features. Which would mean they could shift resources from new features to infra.

"This work" which is being "accelerated" in GP's quote is "the work underway to improve GitHub’s reliability". It is not new features.

Re: The August 17 outage

#644

Post by afc »

ChrisMarshallNY wrote: post #642
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

> this complex system collapses (rather than degrade gracefully) when demand exceeds capacity This. It's easy to be an "armchair quarterback," here, though. Handling stuff like this, needs to be planned for, from the start. I suspect that a lot of the issues are because GitHub is something that started small (and probably quickly), and has accreted. Things like Facebook are in a similar boat.

Haha, tell me about it! I spent more than a decade mostly just deploying these systems across just one company. It's far from trivial!

Re: The August 17 outage

#645

Post by _zoltan_ »

yread wrote: post #512
brookst wrote: post #462

Earlier quoted context omitted.

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

Other projects get unmaintained with maintainers burnt out by a torrent of vulberability reports

usual crying from the usual people. AI slop, blahblahb, ...

(I don't mean you. just these so called open source developers.)

Re: The August 17 outage

#646

Post by kevincox »

k33P1Tr3aL wrote: post #29
jdm2212 wrote: post #7

> Errors in those services triggered a client-side retry loop that increased traffic during recovery. The worst outages I've been part of always have some version of this :(

the 'ol thundering herd problem...

Thundering herd is different. Thundering herd is when a lot of clients trigger requests at the same time. Common situations being a specific time, some other event just occurred or synchronize on other parts of your infrastructure (such as readers queuing behind a R/W lock that then get unblocked at the same time to continue to make a bunch of requests at the same time.

Request amplification via retries is a different problem that causes large amounts of traffic (but it is generally more steady than spiky)

Re: The August 17 outage

#647

Post by afc »

concerned_user wrote: post #611
xnorswap wrote: post #606

Earlier quoted context omitted.

It completely collapsed, 20% is meaningless, as I said in: https://news.ycombinator.com/item?id=49333089 If you load an issue page, you'll see 1 failed request to: /project/product/issues/ And sure, that's what you care about, but consider the working requests to: /in-product-messaging/copilot-budget-request-banner /in-product-messaging/code-scanning-ai-findings-preview-banner /github-copilot/chat /_private/browser/s…

If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.

+1. This is a very real problem in practice.

A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).

We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).

Re: The August 17 outage

#648

Post by 0x696C6961 »

teiferer wrote: post #549
0x696C6961 wrote: post #511

Earlier quoted context omitted.

This is anecdotal, but I know that a lot of my coworkers and coasting and putting up one AI generated PR per day which they've hardly even self reviewed.

In a good company that will come back to bite them next performance review. So, if it doesn't then you learned something about your workplace (and it's not good).

These individuals are demonstrating that they can be replaced with AI. I worry for them.

Re: The August 17 outage

#649

Post by bodge5000 »

hirvi74 wrote: post #382
mey wrote: post #308

Earlier quoted context omitted.

What do you use now and what is your primary use case?

I always advocate for the "boomer" editors when possible. It doesn't matter which one you choose. People can say what they want about them, but very little software survives almost 60 years. I sure as Hell hope VS Code doesn't.

For me, boomer editors (mostly nvim to be honest) join a long list of other things like tiling window managers as something I genuinely want to learn, but that I can never justify learning over either learning something else or doing "real work". I've been slowly introducing both and making gradual progress, but its extremely slow

Re: The August 17 outage

#650

Post by randusername »

neya wrote: post #602
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

I read the article and found it interesting.

It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.

Post reply on HN