Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Git Operations Are Down

githubstatus.com

Post reply on HN
Print view View original

171–180 of 271 posts

Re: GitHub Git Operations Are Down

#171

Post by jcims »

st3fan wrote: post #140
jcims wrote: post #88

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise. https://www.federalregister.gov/documents/2023/06/09/2023-12... On the surface a product like…

GitHub not being available for 45 minutes is not an "extended outage".

Agreed, that's why i said 'tangential at best'.

Re: GitHub Git Operations Are Down

#172

Post by citizenpaul »

hamandcheese wrote: post #153
dangoldin wrote: post #139

Earlier quoted context omitted.

Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.

Systems like these should have an escape hatch of some sort. The key part is that it needs to be auditable. Anything you do in CI should be possible outside of CI, at least by some subset of users.

I've seen numerous "escape hatches" over the years that actually just turned out to be painted on the wall. If you actually tried to use them. No one ever does though.

I don't think its malice. I just think its pretty uncommon for anyone to intentionally back out of a structural tech decision so it gets forgotten about and remains un-battle tested.. That or the timeline is longer than SaaS has been around.

Re: GitHub Git Operations Are Down

#173

Post by spacedcowboy »

freewizard wrote: post #62

It was down for ~2 hours. The status website claims "degraded performance", but in reality we get git@github.com: Permission denied (publickey). Either GitHub didn't know how to communicate, or they were not sure about the real impact. This is bad.

Yeah, I've been trying to check out the SDL3 suite for the last hour or so, and it's still failing on SDL_ttf...

fatal: clone of 'https://github.com/libsdl-org/freetype.git' into submodule path '~/src/SDL/SDL_ttf/external/freetype' failed

...

fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output

...

fatal: clone of 'https://github.com/libsdl-org/harfbuzz.git' into submodule path '~/src/SDL/SDL_ttf/external/harfbuzz' failed Failed to clone 'external/harfbuzz'. Retry scheduled

...

Failed to clone 'external/freetype' a second time, aborting

For a problem that's supposedly "fixed" that's a whole lot of errors...

Re: GitHub Git Operations Are Down

#174

Post by NautilusWave »

itsjustjordan wrote: post #22

I was going insane doubting my SSH knowledge, stopped short of creating new keys thankfully!

I had just upgraded to Windows 11 last week, and for some godforsaken reason earlier today, I could SSH via WSL but not from the host OS even though they were both using keys served from the Windows OpenSSH agent! I'm just going to blame this service outage and hope for the best tomorrow.

Re: GitHub Git Operations Are Down

#175

Post by Over2Chars »

hamandcheese wrote: post #146
Over2Chars wrote: post #141

Earlier quoted context omitted.

I don't see how self-hosting solves the problem of 3rd party vendors unless you're standing up a self-hosting solution as a hot/cold backup to your 3rd party vendor "in the event of an extended outage". Kinda eliminates all those pennies saved (in theory) for outsourcing to "the cloud" if you have to duplicate your infra. Hybrid has always seemed the most optimal approach, but there's always someone in charge who thi…

Since git is distributed, I wonder if it's enough to demonstrate the capability to spin up an alternative, but not necessary keep it up as a live backup 24/7.

If all you care about is insurance, I think you can sometimes merely attest "yeah, it will work" and check a box. No demo necessary (sadly).

If you actually care about uptime, then a real demo with usage is likely the better approach: switch over to your "backup" on a regular basis and make sure it works 100% as expected.

Re: GitHub Git Operations Are Down

#176

Post by theteapot »

Over2Chars wrote: post #163
theteapot wrote: post #144

Earlier quoted context omitted.

Doesn't it exactly solve "extended outages and forced exits from third party vendors."?

Self hosting as an alternative (excluding the 3rd party) or self-hosting as a method of redundancy (in addition to the third party)? If you self host as an ALTERNATIVE to the 3rd party you have all of the same problems - more because you know about them, and the 3rd party can make all these claims you can't verify until they fall over with a "load balancer misconfig" story you also can't verify. If you self-host redu…

I believe the point was protecting against risk of 3rd party falling over. If you self host that reduces the risk. At least you have visibility into the falling over process.

Re: GitHub Git Operations Are Down

#177

Post by sixo »

cyanydeez wrote: post #42
sixo wrote: post #19

Earlier quoted context omitted.

people could not be more clear that their preference is for reliable and easy-to-use centralized services maintained by professionals, and not decentralized systems that require a great deal of user expertise

Thats mostly because of herdmentality not because of a considered approach. Decentralization assumes the majority are anti social.

No, it's because complexity comes with a cognitive cost, and delegating responsibility to other entities minimizes this

If you don't account for the benefit, it looks irrational, but this is true of absolutely anything

Re: GitHub Git Operations Are Down

#178

Post by Over2Chars »

theteapot wrote: post #176
Over2Chars wrote: post #163

Earlier quoted context omitted.

Self hosting as an alternative (excluding the 3rd party) or self-hosting as a method of redundancy (in addition to the third party)? If you self host as an ALTERNATIVE to the 3rd party you have all of the same problems - more because you know about them, and the 3rd party can make all these claims you can't verify until they fall over with a "load balancer misconfig" story you also can't verify. If you self-host redu…

I believe the point was protecting against risk of 3rd party falling over. If you self host that reduces the risk. At least you have visibility into the falling over process.

It reduces the risk only if your self-hosted solution also doesn't fall over.

It's like saying "I can reduce the risk of my rental car failure by owning my own car", assuming your own car you keep undriven in your garage, doesn't have a dead battery, no gas, flat tires, and proves to be unusable for hauling.

The "cloud" was touted as the fix for all that nuisance in self-hosting. Magically Jeff's bit barn would work to five 9s of uptime, and you could sit back and write your code, unshackled to infra. Until Jeff's bit barn went tits up.

I say the "cloud" is just another guys data center behind an API.

You wanna cloud experience? Put an API in front of your own servers and burn a $100 bill.

Re: GitHub Git Operations Are Down

#179

Post by hamandcheese »

dangoldin wrote: post #167
hamandcheese wrote: post #153

Earlier quoted context omitted.

Systems like these should have an escape hatch of some sort. The key part is that it needs to be auditable. Anything you do in CI should be possible outside of CI, at least by some subset of users.

Yea - definitely. Just not ideal and something that needs to be built out, tested, etc.

Yes, it is easier said than done. At my company we use buildkite and many people wrote scripts that simply fail outside of buildkite.

GitHub actions is even worse, it seems like it was designed from the ground up to create lock in.

Nix helps a bit on the bootstrapping and dependency management problem, but won't save you from writing a script that is too tightly coupled to its runtime environment.

Re: GitHub Git Operations Are Down

#180

Post by hamandcheese »

Over2Chars wrote: post #175
hamandcheese wrote: post #146

Earlier quoted context omitted.

Since git is distributed, I wonder if it's enough to demonstrate the capability to spin up an alternative, but not necessary keep it up as a live backup 24/7.

If all you care about is insurance, I think you can sometimes merely attest "yeah, it will work" and check a box. No demo necessary (sadly). If you actually care about uptime, then a real demo with usage is likely the better approach: switch over to your "backup" on a regular basis and make sure it works 100% as expected.

My hypothetical universe is "I believe GitHub is too big to fail and want to spend as little resources to please the auditor as is reasonably possible without resorting to fraud".

So really what I'm asking is "how strict are these audits really?"

Post reply on HN