Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

241–250 of 804 posts

Re: The August 17 outage

#241

Post by mikert89 »

jeremyjh wrote: post #217
mikert89 wrote: post #179

Earlier quoted context omitted.

AWS saw this growth every year for two decades, hyper growth tech sees it all the time

The absolute number doesn’t matter nearly as much as the change in rate of growth. The number of commits had not been doubling every six months at GitHub for a long time. At Amazon if traffic volume consistently doubled every six months that is actually quite a lot easier to plan for, it just becomes part of everything they do from very early on.

im not buying that github is a unique engineering problem, harder than the rest of hyperscalars.

Re: The August 17 outage

#242

Post by dcrazy »

ferngodfather wrote: post #233
kylecazar wrote: post #158

Earlier quoted context omitted.

GH processes at the commit level for things (including actions) even though they're bundled in a push... it's relevant to the load on their infrastructure.

But they can't process the commit until I push it?

A push containing 100 commits is more expensive than a push containing 1 commit.

Re: The August 17 outage

#243

Post by sqquima »

stingraycharles wrote: post #218
taylor-s wrote: post #211

Earlier quoted context omitted.

Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.

Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering. Source: decades of operational pain.

This is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems. It's also true that we're witnessing unprecedented levels of scale.

Re: The August 17 outage

#244

Post by fragmede »

b112 wrote: post #133
Macha wrote: post #125

Earlier quoted context omitted.

Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then: If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway. If alternatives aren't able to manage, then user's aren't going to jump since those services won't actua…

I think the number of commits is a red herring, but that said, I wonder how spiky their load is. Imagine a fee over X commits, but only during certain hours. I can imagine 90% of the commits over 6 or 8 timezones, maybe 50% over 4 right now...

It's not a red herring, every commit necessitates a database hit.

Re: The August 17 outage

#245

Post by Olreich »

Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

https://brooker.co.za/blog/2022/02/28/retries.html

Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful

Re: The August 17 outage

#247

Post by dannyw »

jdm2212 wrote: post #200
jordanb wrote: post #194

Earlier quoted context omitted.

Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR

Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.

Your highly available system is probably somewhat important, otherwise you won’t have invested in making it HA.

While your premise holds for happy cases, when you do have a cascading series of outages, not using exponential backoff is just adding a self-inflicted DoS to when you do go down.

I don’t really follow your premise and can’t really articulate many cases for when you shouldn’t use exponential backoff. Maybe if you’re working at Jane St or something; or other circumstances where you can deploy immediate changes to the client; and you’re willing to trade ‘better p50 for worse outages’.

But in the case of shipped code that’s run on clients, I’ll continue exponentially backing off all the way, all the time, for everything.

Re: The August 17 outage

#249

Post by MeetingsBrowser »

Quarrelsome wrote: post #214
taylor-s wrote: post #211

Earlier quoted context omitted.

Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.

isn't that a bomb with a pair of scissors to cut the fuse that could break down under certain conditions? I feel like they could also hide an issue that might get fixed if there were no retries. Is it slow or is our resource sporadically offline?

Exponential back off retries could hide a real issue, but I would estimate something like 99.99999999% of network retries are resolved within the first 2 attempts.

Not using retries is optimizing for the astronomically rare case, which is better mitigated by other means

Re: The August 17 outage

#250

Post by TomBombadildoze »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

[flagged]
Post reply on HN