Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

211–220 of 804 posts

Re: The August 17 outage

#211

Post by taylor-s »

Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.

Re: The August 17 outage

#212

Post by bbarn »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.

Re: The August 17 outage

#213

Post by wat10000 »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

Test coverage varies a lot, but error paths seem almost universally untested.

Re: The August 17 outage

#214

Post by Quarrelsome »

taylor-s wrote: post #211
Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.

isn't that a bomb with a pair of scissors to cut the fuse that could break down under certain conditions?

I feel like they could also hide an issue that might get fixed if there were no retries. Is it slow or is our resource sporadically offline?

Re: The August 17 outage

#215

Post by wat10000 »

jdm2212 wrote: post #200
jordanb wrote: post #194

Earlier quoted context omitted.

Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR

Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.

The whole point of exponential backoff is that the first retry can be quick.

Re: The August 17 outage

#216

Post by pixl97 »

r3trohack3r wrote: post #113
Aachen wrote: post #93

Earlier quoted context omitted.

I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference. Do you know of a documented instance w…

Have experienced it, but didn’t document. Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy. Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microser…

Exactly, without jitter the thundering heard problem turns into trying to escape a small island with a small boat and big waves hitting the shore problem. You can never fully recover before you get smashed again.

Re: The August 17 outage

#217

Post by jeremyjh »

mikert89 wrote: post #179
tracerbulletx wrote: post #90

Earlier quoted context omitted.

When a service that was already the primary git hosting provider for most of the world for 20 years grows at that rate its not mundane and its not comparable to any example.

AWS saw this growth every year for two decades, hyper growth tech sees it all the time

The absolute number doesn’t matter nearly as much as the change in rate of growth. The number of commits had not been doubling every six months at GitHub for a long time.

At Amazon if traffic volume consistently doubled every six months that is actually quite a lot easier to plan for, it just becomes part of everything they do from very early on.

Re: The August 17 outage

#218

Post by stingraycharles »

taylor-s wrote: post #211
Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.

Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering.

Source: decades of operational pain.

Re: The August 17 outage

#219

Post by pseudosavant »

blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

For some context, in June they said commits "commits nearly doubled year over year, crossing 1.4 billion per month". Now, it has more than doubled that in just a few months.

https://github.blog/news-insights/product-news/github-copilo...

Re: The August 17 outage

#220

Post by dadadad100 »

Aachen wrote: post #93
r3trohack3r wrote: post #75

Earlier quoted context omitted.

Don’t forget jitter!

I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference. Do you know of a documented instance w…

It’s a fundamental property of Ethernet[1]. Does that count?

[1] https://en.wikipedia.org/wiki/Exponential_backoff#Example

Post reply on HN