Live data from Hacker News

GitHub appears to be struggling with measly three nines availability

theregister.com

161–170 of 255 posts

Re: GitHub appears to be struggling with measly three nines availability

#161
post #113

I don't want to give too much credit to Github, because their uptime is truly horrendous and they need to fix it. But: I've felt like its a little unfair to judge the uptime of company platforms like this; by saying "if any feature at all is down, its all down" and then translating that into 9s for the platform. I never use Github Copilot; it does go down a lot, if their status page is to be believed; I don't really…

You're right that labelling any outage as "Github is down" is an overgeneralisation, & we should focus on bottlenecks that impact teams in a time sensitive matter, but that isn't the case here. Their most stable service (API) has only two 9s (99.69%). They're not even struggling to get their average to three 9s, they're struggling to get ANY service to three 9s. They're struggling to get many services to two 9s. Copi…

Nine nines is too hard; my target is eight eights.

Re: GitHub appears to be struggling with measly three nines availability

#162

Earlier quoted context omitted.

> I've felt like its a little unfair to judge the uptime of company platforms like this; by saying "if any feature at all is down, its all down" and then translating that into 9s for the platform. This is definitely true. At the same time, none of the individual services has hit 3x9 uptime in the last 90 days [0], which is their Enterprise SLA [1] ... > "Uptime" is the percentage of total possible minutes the applica…

So what happens for those enterprise customers now? Is there a meaningful fallout when these services fail to meet their SLAs?

> If GitHub does not meet the SLA, Customer will be entitled to service credit to Customer's account ("Service Credits") based on the calculation below ("Service Credits Calculation").

The linked document in my previous comment has more detail.

Re: GitHub appears to be struggling with measly three nines availability

#163
post #18

While GitHub obsess over shoving AI into everything, the rest of the platform is genuinely crumbling and its security flaws are being abused to cause massive damage. Last week Aqua Security was breached and a few repositories it owns were infected. The threat actors abused widespread use of mutable references in GitHub Actions, which the community has been screaming about for years, to infect potentially thousands of…

If you want more ammo for your ranting (no offense meant, I also rant): an issue as massive as https://github.com/orgs/community/discussions/142308 lingering for years should do the trick.

* no ipv6 support

* no ff merge support

* no sha2 commit hash support

Re: GitHub appears to be struggling with measly three nines availability

#164
post #113

I don't want to give too much credit to Github, because their uptime is truly horrendous and they need to fix it. But: I've felt like its a little unfair to judge the uptime of company platforms like this; by saying "if any feature at all is down, its all down" and then translating that into 9s for the platform. I never use Github Copilot; it does go down a lot, if their status page is to be believed; I don't really…

It doesn't help that almost all of the big tech companies talking about 5 9s are lying about it; "Does it respond to the API at all, even with errors? It's up!" and so on. If you spend a lot of time analyzing browser traces you see errors and failures constantly from everyone, even huge companies that brag a lot about their prowess. But it's "up" even if a shard is completely down.

The five nines tech people usually are talking about is a fiction; the only place where the measure is really real is in networking, specifically service provider networking, otherwise it's often just various ways of cleverly slicing the data to keep the status screen green. A dead giveaway is a gander at the SLAs and all the ways the SLAs are basically worthless for almost everyone in the space.

See also all of the "1 hour response time" SLAs from open source wrapper companies. Yes, in one hour they will create a case and give you case ID. But that's not how they describe it.

Re: GitHub appears to be struggling with measly three nines availability

#165
post #113

I don't want to give too much credit to Github, because their uptime is truly horrendous and they need to fix it. But: I've felt like its a little unfair to judge the uptime of company platforms like this; by saying "if any feature at all is down, its all down" and then translating that into 9s for the platform. I never use Github Copilot; it does go down a lot, if their status page is to be believed; I don't really…

There's a completely reasonable comment by jamiemallers on this thread which is marked as 'dead' even after vouching. Not sure what's going on there.

Re: GitHub appears to be struggling with measly three nines availability

#166
post #113

I don't want to give too much credit to Github, because their uptime is truly horrendous and they need to fix it. But: I've felt like its a little unfair to judge the uptime of company platforms like this; by saying "if any feature at all is down, its all down" and then translating that into 9s for the platform. I never use Github Copilot; it does go down a lot, if their status page is to be believed; I don't really…

There's a completely reasonable comment by jamiemallers on this thread which is marked as 'dead' even after vouching. Not sure what's going on there.

Take a look at his comment history.

Re: GitHub appears to be struggling with measly three nines availability

#167
post #100

Earlier quoted context omitted.

The problem is actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 probably doesn’t do this same pinning, and the actions ecosystem is such an intertwined mess that any single compromised action can propagate to the rest

Yes, true, but at least the fire won't spread through this one point. Hopefully all of your upstreams can be persuaded to pin also.

Doesn't a single compromised action in the chain cause the whole to be fucked? Pinning the top level doesn't prevent any spread.

Re: GitHub appears to be struggling with measly three nines availability

#169

Earlier quoted context omitted.

Note that it gets 90% largely off Copilot going down and Actions not working. Actual git has 98.98%, which is still just one 9 but a lot better.

> 98.98% it's the 2 nines they aimed for

True! Technically even 9.99% would be three nines!

Re: GitHub appears to be struggling with measly three nines availability

#170
To be honest, I’m not surprised that GitHub has been having issues.

If you have ever operated GitHub Enterprise Server, it’s a nightmare.

It doesn’t support active-active. It only supports passive standbys. Minor version upgrades can’t be done without downtime, and don’t support rollbacks. If you deploy an update, and it has a bug, the only thing you can do is restore from backup leading to data loss.

This is the software they sell to their highest margin customers, and it fails even basic sniff tests of availability.

Data loss for source code is a really big deal.

Downtime for source control is a really big deal.

Anyone that would release such a product with a straight face, clearly doesn’t care deeply about availability.

So, the fact that their managed product is also having constant outages isn’t surprising.

I think the problem is that they just don’t care.

Post reply on HN