Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

691–700 of 804 posts

Re: The August 17 outage

#691

Post by inigyou »

jdm2212 wrote: post #200
jordanb wrote: post #194

Earlier quoted context omitted.

Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR

Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.

Someone linked the Google SRE book. It explains that client services watch the global distribution (for that client) of number or retries. It will normally retry immediately on another node, because as you say, some failures are expected. But if it notices that more than about 1% of requests are having to retry twice, that indicates the server service needs reduced load and it starts refusing to retry. If it gets really bad it even starts preemptively failing first attempts so that the service doesn't get contacted at all.

Re: The August 17 outage

#692

Post by yard2010 »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Thank you so much for this gem, this sent me through the rabbit hole and I'm fascinated by the level of engineering in this book. I love how big complex systems are engineered. It feels like anything is possible when you have a solid plan and you keep improving it step by step. So inspiring.

Re: The August 17 outage

#693

Post by inigyou »

27183 wrote: post #450
MrWiffles wrote: post #406

Earlier quoted context omitted.

We used to be able to afford two! Acronym letters cost as much as houses nowadays!

OK I'll pick C and A! https://web.archive.org/web/20250128235041/https://codahale....

The problem is that P is actually part of A. It should be called the CA theorem.

Re: The August 17 outage

#694

Post by _kidlike »

cube00 wrote: post #664
_kidlike wrote: post #471

Earlier quoted context omitted.

So, in all of your software, you have introduced randomness in your retries so that the billions of your clients avoid retry synchronization dances?

I'm not asking for randomness, I'm only asking it to stop retrying after say 10 seconds and report it to the user as a fatal error.

it would be surprising if this wasn't already the case.

Re: The August 17 outage

#695

Post by inigyou »

hirvi74 wrote: post #398
bluerooibos wrote: post #302

Earlier quoted context omitted.

> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour That wouldn't be a unit test - that's more like an end-to-end or integration test. Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.

My employer has 0% test coverage lol. I've begged and pleaded, but the claim is that "risk is low" and "that's what QA is for." Hell, I've complained to senior management about how there are senior devs that forgo backend validation. It's truly Hell in the trenches sometimes. Some days, I would seriously rather work at Wendy's.

Just write the tests. It's your job, not your employer's.

Re: The August 17 outage

#696

Post by 13639366668 »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

[flagged]

Re: The August 17 outage

#697

Post by inigyou »

Lammy wrote: post #321
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

> a wider trend to avoid showing the user any error at all costs And in fact you can see the degradation of software over the previous decade-plus via Google Trends search for ‘something went wrong’ lol: https://trends.google.com/trends/explore?date=all&q=%22Somet...

Ironically I get an error page when I click on this.

Re: The August 17 outage

#698

Post by inigyou »

chasd00 wrote: post #361
giancarlostoro wrote: post #235

Earlier quoted context omitted.

Backend API rate limiting has to surely kick in and force you to wait x amount of time before you try again… Discords bot API actually sends you how long before you retry.

As a client you can hit a server as much and as often as you like. The only thing the server can do is return an error code or try to hold the socket open (which the client can then close on their own).

It can block your IP address at the firewall level. It can't stop you DDoSing it with raw packets, but that's very unlikely to happen unintentionally, because TCP will wait for a SYN-ACK response for at least several seconds and possibly up to several minutes.

Re: The August 17 outage

#699

Post by inigyou »

sien wrote: post #312
TomBombadildoze wrote: post #250

Earlier quoted context omitted.

[flagged]

Ironically wouldn't someone who solves real problems at scale actually be a 'keyboard warrior'?

Ironically, yes, they fight using a keyboard. But not in the way that the term "keyboard warrior" means.

Re: The August 17 outage

#700

Post by conductr »

otterley wrote: post #480
conductr wrote: post #152

Earlier quoted context omitted.

Conversely, what suggests GitHub has a huge operating cost? Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right? It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitH…

> what suggests GitHub has a huge operating cost? They just added 3 million CPU cores to the fleet. That’s a lot of cheddar.

Absolute numbers aren’t useful in this analysis. We know capital is easily available for the expenditure. You have to consider it as a relative number to revenue or users otherwise big numbers always look scary.
Post reply on HN