Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

231–240 of 804 posts

Re: The August 17 outage

#231

Post by kloop »

muglug wrote: post #178
sandeepkd wrote: post #172

Earlier quoted context omitted.

Not sure to be honest, from a machine perspective 2X should never be a big deal, unless 1.4 was the threshold or sweet state and no one thought too much about scale and architecture beyond that

If you’re the size of GitHub and you’ve been running your infra for years with very little variation in traffic patterns you have a strong incentive to optimise costs for that existing behaviour.

Sure, but the new patterns are several years old at this point.

Not adapting is a choice

Re: The August 17 outage

#232

Post by dannyw »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.

Re: The August 17 outage

#233

Post by ferngodfather »

kylecazar wrote: post #158
tremon wrote: post #145

Earlier quoted context omitted.

Why is Github talking about number of commits here, and not pushes? Are there a lot of tools/people using github as an online editing platform?

GH processes at the commit level for things (including actions) even though they're bundled in a push... it's relevant to the load on their infrastructure.

But they can't process the commit until I push it?

Re: The August 17 outage

#234

Post by mananaysiempre »

0xblinq wrote: post #42
cschep wrote: post #17

Earlier quoted context omitted.

https://forgejo.org/ promises to be this, have only lightly used it on https://codeberg.org/ but it seems nice?

"Forgejo is a self-hosted lightweight software forge" That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous. I still don't know what it is or what it does.

“Software forge” is an established[1] term by this point: a piece of software providing a full set of features for collaborative development, usually at least code hosting with a web interface, tickets, and webpage hosting, and these days often also code review and integrated CI or at least the possibility of integrating CI. It’s admittedly squishy but in the same way “IDE” is squishy, which is to say it’s still well-defined enough to be useful.

[1] https://en.wikipedia.org/wiki/Forge_(software)

Re: The August 17 outage

#235

Post by giancarlostoro »

cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

Backend API rate limiting has to surely kick in and force you to wait x amount of time before you try again… Discords bot API actually sends you how long before you retry.

Re: The August 17 outage

#236

Post by yoyohello13 »

jdm2212 wrote: post #22
annoyingnoob wrote: post #15

Github down, no hard drives available, no memory available, thanks AI! Seems like we are headed for Tech Gridlock.

This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.

Layoffs by the 10s of thousands, food prices out of control, people barely able to afford gas. At least some tech bros can launch their 50th B2B SaaS. I didn't realize a booming economy sucked so much.

Re: The August 17 outage

#237

Post by jdm2212 »

wat10000 wrote: post #215
jdm2212 wrote: post #200

Earlier quoted context omitted.

Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.

The whole point of exponential backoff is that the first retry can be quick.

The right answer is for the RPC framework to accurately communicate "try again on another node" vs "don't try again, just hard fail".

When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will time out before you finish all the work, at which point you're doing a bunch of slow expensive work for no gain (and the enqueued slow expensive work will make your outage worse).

Re: The August 17 outage

#238

Post by Olreich »

cube00 wrote: post #228
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?

It's less about "can't afford" and more about "don't want to spend".

Re: The August 17 outage

#239

Post by ethin »

Is it me or is all of this essentially "we don't want to show the user anything at all when something breaks?"

And what makes this funny (to me) is that this is a website for developers. I would think that of all the audiences you would target, developers would mind seeing the platform display error messages when things break the least.

Re: The August 17 outage

#240

Post by beyonddream »

cube00 wrote: post #228
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?

Answer is, you do comprehensive unit testing irrespective of Microsoft or any other company doing it. Also, as the OP of your comment posted, you are over estimating the software quality of these big name corporate companies. It can’t be the bar against which you would want measure your own company standards.
Post reply on HN