Live data from Hacker News

The August 17 outage

github.blog

131–140 of 804 posts

Re: The August 17 outage

#131
post #83
post #53

Earlier quoted context omitted.

A server box now has around 256 CPU cores. So that's about 12000 servers. If each one is $10k that's $120 million. Not a lot compared to Github's income.

$10,000 is only enough for CPU cost (2x AMD EPYC 9754). The rest of the servers (mainboard, RAM, storage, power supply, network card, rack, cooling, etc.) can easily triple or quadruple the cost.

Yup. Still, it's less than their annual income by several times.

Re: The August 17 outage

#132
> Errors in those services triggered a client-side retry loop that increased traffic during recovery

Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.

> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.

Re: The August 17 outage

#133
post #125

Earlier quoted context omitted.

That would just drive users into the arms of the alternatives, which would love to own the world's code... like Cursor/Musk. Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue. As someone on the sidelines, this is really interesting to watch unfold.

Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then: If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway. If alternatives aren't able to manage, then user's aren't going to jump since those services won't actua…

I think the number of commits is a red herring, but that said, I wonder how spiky their load is.

Imagine a fee over X commits, but only during certain hours. I can imagine 90% of the commits over 6 or 8 timezones, maybe 50% over 4 right now...

Re: The August 17 outage

#134
post #44

Earlier quoted context omitted.

Exponential backoff is your friend... too few people use it.

Don’t forget jitter!

Reminds me of this blog post I read yesterday, about Buildkite introducing jitter in their CI agent to reduce synchronisation

https://buildkite.engineering/sleeping-at-scale/

Re: The August 17 outage

#135
post #77

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

Not that impressive when you realize it's mostly due to AI slop edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.

I don't know how to delete things.

Re: The August 17 outage

#136
post #87

Earlier quoted context omitted.

Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).

How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person

Yeah, seems like AI slop is going to kill GitHub's free tier. I just don't see how the economics of having to host this much slop and provide service to slopcoders is going to convert into dollars for them otherwise. None of the humans involved are going to end up in big enterprises. It's all cost, with no pathway to revenue.

Re: The August 17 outage

#137
post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

Except the other side of this is interrupting a service which would otherwise have succeeded: there's a lot of unattended or minimally attended processes where an interruption is just asking the user to do the only thing they were going to do anyway - retry it.

In GitHub's case this is especially relevant - the only reason to throw an error message at the user is the hope they - the human - give up and walk away (or you break all the CI/CD builds and the time it takes humans to hit "retry" gives you some breathing room).

Re: The August 17 outage

#139

Earlier quoted context omitted.

Not really. It's AI commits. Not quality commits.

it doesn't matter though, commits create load in their system

One has to wonder what the point of a collaborative version control system even is if the software we’re writing is one-shotted by a call to an API ?

Re: The August 17 outage

#140
post #96
post #39

Earlier quoted context omitted.

I'm betting on Tangled and Codeberg. Tangled has a better press and in general is a dark horse, Codeberg has the "brand" and some network effects from projects that moved to there. (famously, Zig.) I heard that Sourcehut is having a moment as well, and I love the idea of email-based workflow and not having to have an account to contribute to someone's project hosted there, but I'm not maintaining anything worthwhile…

What if it was $2?

that is more manageable but c'mon I can't even keep Google One 100GB up on a consistent basis, that's how poor I am. self-hosting would be a far better option because apparently I find enough people to provide free Hetzner VPSes and stuff as long as I can sell this as mutually beneficial.

for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail

EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option

EDIT 2: the email part would be PITA, so $4/mo is attractive on that background

Post reply on HN