Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

361–370 of 804 posts

Re: The August 17 outage

#361

Post by chasd00 »

giancarlostoro wrote: post #235
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

Backend API rate limiting has to surely kick in and force you to wait x amount of time before you try again… Discords bot API actually sends you how long before you retry.

As a client you can hit a server as much and as often as you like. The only thing the server can do is return an error code or try to hold the socket open (which the client can then close on their own).

Re: The August 17 outage

#363

Post by stillpointlab »

arn3n wrote: post #78

Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI. I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.

They could do this in a way that lessens external pull requests.

For example, bolting co-pilot on to github, or a Codex in the web kind of thing that gives unlimited check ins.

It's like how Grok Heavy gives the user X premium or whatever. You charge for the tokens and give the unlimted premium access as a bonus. Basically, bundle it.

Re: The August 17 outage

#365

Post by nickelpro »

StilesCrisis wrote: post #184

"... these incidents make clear that we must accelerate this work." It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.

Do you think the load is going away?

The current infrastructure cannot handle the new load requirements. Either the infrastructure must change, or they must start denying users the ability to use the infrastructure.

Re: The August 17 outage

#366

Post by anonymars »

Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

You're not wrong - https://devblogs.microsoft.com/oldnewthing/20051107-20/?p=33...

Re: The August 17 outage

#367

Post by NameError »

The 'growth in completed actions runs' graph is interesting. I assume the periodic drops are weekends, so intuitively the floor of those drops corresponds more with hobby/personal projects than people at work. It looks like there's a sharp uptick specifically in that floor since July ish.

Re: The August 17 outage

#368

Post by Brian_K_White »

ninkendo wrote: post #304
leptons wrote: post #182

Earlier quoted context omitted.

>You can rig up a local ide to pathologically commit+push per save The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder…

> requires a commit and push for every file save I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.

meh, it's just external undo button. It's useful. It might or might not be worth the cost, but it's not that it delivers no value or causes some harm (other than cost/reliability)

Re: The August 17 outage

#369

Post by Brian_K_White »

sajithdilshan wrote: post #62

[flagged]

Whos fault is the massive load increase, or their inability to navigate it?

Did I iject copilot into every part of github? In fact not only did I not insert it, I have never even used it.

Did I move their infrastructure to Azure?

Did I sell them to MS?

Did I set all the directives and priorities that MS has set on them like telling everyone they must use openai for everything, and then telling them they must stop doing that and user their own ai instead?

The outage is not due to a natural disaster that no one could anticipate and no one had any input on creating the conditions. They keep the free tier because THEY want what THEY get from the free tier. They could easily have a $1 tier and various totally sensible throttle limits on various services and apis that would have avoided all this, but that would not get them the 100% user coverage that they want. So THEY choose to provide free, swiss cheese service.

It's not some unreasobable burden they labor under that anyone else should be understanding and forgiving about.

Re: The August 17 outage

#370

Post by aprdm »

Breakthrough wrote: post #357
lbrandy wrote: post #283

Earlier quoted context omitted.

I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.

You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?

Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
Post reply on HN