Live data from Hacker News

The August 17 outage

github.blog

711–720 of 804 posts

Re: The August 17 outage

#711

Earlier quoted context omitted.

isn't that a bomb with a pair of scissors to cut the fuse that could break down under certain conditions? I feel like they could also hide an issue that might get fixed if there were no retries. Is it slow or is our resource sporadically offline?

Google “thundering herd” and you’ll understand why uncontrolled retries can be / are bad.

Quarrelsome is suggesting you should have no retries, not unlimited.

Re: The August 17 outage

#712

Earlier quoted context omitted.

I totally agree with you, I think retries are overused, with the exception of operations that are known to be unreliable and can't be improved. In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing. I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causin…

I have the exact opposite view. Way too often, I’ll be presented with an error to the effect of, “something went wrong, please try again” and often the retry works. And I’m left wondering why this machine whose sole purpose is to automate things can’t do that for me automatically. In particular, networks tend to be a LOT less reliable than the typical developer accounts for. And the failures are very often transient.…

If you want a retry to prevent that, it must only be done at the outermost layer. Note that what you think is the outermost layer might not be, and that accidentally deploying a retry at a non-outermost layer is much worse than no retry at all.

Re: The August 17 outage

#713

Linux didn't (yet) kill Microsoft. Microsoft absorbed that shot. Then the Git arrow went straight to cold black heart of Microsoft. The next few months will determine if they survive it. If they do, what will we see from the third draw out of Linus' quiver?

Linux had nothing to do with the downfall of Microsoft. Any perceived dominance of Linux is not because Linux is getting better, but because Microsoft is getting worse.

Re: The August 17 outage

#714
post #661

"The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR " - silly question but github uses github for their own PRs and deploy right? Do they have a special dedicated system just for them so they can fix github with a code change even if the rest of us can't?

Not sure, but they could use the self-hosted enterprise version of GitHub for their use case.

Re: The August 17 outage

#715
post #707

Earlier quoted context omitted.

The problem is that P is actually part of A. It should be called the CA theorem.

It's all about what you choose in the face of P. Do you choose C or do you choose A?

Yes. But calling it CP or AP makes it seem like you can also choose CA.

(Also both CP and AP have alternative very dodgy meanings)

Re: The August 17 outage

#716

I wonder how much of the traffic increase is enterprise vs. hobbyists? A 7 hour outage for enterprise customers is really, really bad and it's sad if caused by a mass of non-paying vibe coders. It's becoming absolutely obvious that the unlimited free tier needs to go.

This is my suspicion, all though I have no evidence, just anecdotes. I make a handful of commits a day. I write code, review it, test it, commit, and then push.

We have some marketing folks that have gotten into vibe coding stuff for their personal use. First let me say: good for them and I'm glad they're experimenting with new ideas and tools. The side effect of that is that looking at their repos, they're having Claude go whole hog and make upwards of hundreds of commits a day, all with things that they haven't taken a look at.

I don't think I can say this is wrong of them, because their tools encourage that and they shouldn't have to consider their impact on an enterprise service, but I wonder if this trend is similar in other places.

Re: The August 17 outage

#717
post #327

I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free. I've been saying this for a while: https://news.ycombinator.com/item?id=47534499

Maybe GitHub sees value in hosting the code that is produced, for model training purposes. This would justify to maintain a free offering, even when it gets flooded like it has been recently

Re: The August 17 outage

#718
post #327

I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free. I've been saying this for a while: https://news.ycombinator.com/item?id=47534499

Maybe GitHub sees value in hosting the code that is produced, for model training purposes. This would justify to maintain a free offering, even when it gets flooded like it has been recently

And even when this code is AI produced, yes. It has some informational value.

Re: The August 17 outage

#719

Earlier quoted context omitted.

In a good company that will come back to bite them next performance review. So, if it doesn't then you learned something about your workplace (and it's not good).

> In a good company that will come back to bite them next performance review. In a good company that should be discussed in the next 1:1s so actual change can happen meanwhile. If it just waits for the end of year review, then it's not a good company.

Sure. I meant if they are stubborn.

Re: The August 17 outage

#720

Earlier quoted context omitted.

> what suggests GitHub has a huge operating cost? They just added 3 million CPU cores to the fleet. That’s a lot of cheddar.

Absolute numbers aren’t useful in this analysis. We know capital is easily available for the expenditure. You have to consider it as a relative number to revenue or users otherwise big numbers always look scary.

I agree - however, I was responding to your direct question about whether GitHub's operating cost was high, and there's some data that suggests that it is!
Post reply on HN