Live data from Hacker News

The August 17 outage

github.blog

191–200 of 804 posts

Re: The August 17 outage

#191

> Since April, monthly commits have grown from 1.4 billion to 2.9 billion Bonkers. You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.

Cry in story points. Like a real scrum master level 9000.

Re: The August 17 outage

#192
post #127

Earlier quoted context omitted.

No doubt. ...You can have non-zero revenue and still be loss-making though.

Conversely, what suggests GitHub has a huge operating cost? Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right? It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitH…

It used to be, and buying a few servers was a cheap way to get off the cloud (if you can manage them efficiently). But with RAM and other costs these days... I've had to rethink it a lot.

Re: The August 17 outage

#193

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

I was using claude tethered via my phone, and would lose signal every now and then as we went through a tunnel. I was glad for how resilient it was its its eventual retries.

Re: The August 17 outage

#194
post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…

Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR

Re: The August 17 outage

#195

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

I don't like blind retries. It's different if the server or LB knows it's overloaded and asks clients to retry in X seconds.

Re: The August 17 outage

#196

Here's Vladimir Fedorov's GitHub contribution graph, as linked to as the author of this post: https://imgur.com/a/zIbT0Gi It shows zero contributions in the past year, on this account. This is a huge, huge red flag.

I strongly disagree. GitHub, a year ago, acknowledged the fundamental problems and began work on them. We all agree with the diagnosis and strategy: stop building new things, bring stability. Why would whether the CTO codes have any bearing on the correctness of this strategy? GitHub’s problem isn’t that leadership don’t understand the product, or that they don’t know what they should be doing, it’s that they’re batt…

Interesting - There's a couple of things that I disagree with here, but I do think this resonates: "We all agree with the diagnosis and strategy: stop building new things, bring stability."

Where I have a problem with the positioning of their GitHub profile is, he's the CTO of GitHub, arguably the defacto standard for open-source version control systems. His GitHub profile is linked to as the author for the post, and his GitHub profile simply tells me: "this guy doesn't code."

I don't care if this guy doesn't work on GitHub itself, I hardly would expect that, but IMO, any CTO of a company like GitHub should eat, breathe, and sleep code. He might, but his profile, which is being published as if it means anything, tells me he doesn't.

Re: The August 17 outage

#197

Earlier quoted context omitted.

I'm not following your line of questioning. Without ever using github as an online editing platform, you can do one push with two new commits.

Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push. You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.

Do you have some GitHub architectural knowledge you’d like to share with us?

A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit.

There’s a difference between pushing 1 commit and 100.

Re: The August 17 outage

#198

Earlier quoted context omitted.

LOL, this kind of things will happen when projects like Bun ( https://github.com/oven-sh/bun ) are running on auto. :)

3.3k open issues holy shit

Strange to think they are probably triaged by LLMs at this point

Re: The August 17 outage

#199

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

I totally agree with you, I think retries are overused, with the exception of operations that are known to be unreliable and can't be improved.

In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing.

I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causing cascading failures, it is a degraded user experience when it doesn't succeed.

As a user I would rather see an error straight away than see many seconds of spinning while something silently retries, and THEN an error.

Re: The August 17 outage

#200

Earlier quoted context omitted.

A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…

Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR

Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.
Post reply on HN