Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Git Operations Are Down

githubstatus.com

Post reply on HN
Print view View original

251–260 of 271 posts

Re: GitHub Git Operations Are Down

#251

Post by 0xbadcafebee »

Gigachad wrote: post #181
0xbadcafebee wrote: post #112

Earlier quoted context omitted.

Basically you rent space and machines in datacenters that are geographically distributed and designed to resist natural disasters

That's exactly what Github does already. I wouldn't bet on my own org being better at hosting github than github.

Agreed. I have worked for several companies who tried to self-host their VCS etc, and their uptime was way worse, the overall cost was higher, and there were less features. Having a backup is always a good idea though, and an emergency plan for how to onboard to a competitor.

Re: GitHub Git Operations Are Down

#252

Post by 0xbadcafebee »

pnw_throwaway wrote: post #127
0xbadcafebee wrote: post #78

> all Git operations were unavailable due to a configuration change eats popcorn waiting for explanation of why they didn't catch it in non-prod

Have you never fixed a prod-specific bug before? They’re common enough, and it sounds like they patched this one within a reasonable amount of time.

I expect more from billion-dollar companies. If you have that much money, you can pay somebody to ensure the changes are properly tested. (I've worked in billion dollar companies, and usually they are too cheap to do this, despite their massive wealth. They just leave it up to ICs and hope they are testing right, rather than making sure)

Re: GitHub Git Operations Are Down

#253

Post by anbotero »

bgrainger wrote: post #190
thayne wrote: post #189

Earlier quoted context omitted.

Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud. And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable…

We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.) Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infre…

I was the administrator of a GitHub Enterprise Server instance back in 2015-2016 (I think 2014 too).

Rock-solid stability, for a company with 300+ microservices, 10+ big environments, 50+ microenvironments, who knows how many Jenkins pipelines (more than 900, I’ll tell you that). We deployed several times a day, each service on average had 3 weekly deployments.

As a company, I think GitHub (public) should do better, much better, given this is happening more frequently as of late, but if big companies (even medium ones) don’t have their own package caches, they are all in for a ride.

At a previous Startup we had GitHub + GitHub Actions, and we were on AWS. We setup some OCI images cache. Sure, if GitHub went down we could not deploy new stuff, but at least it wouldn’t take us down. If we really needed the pipelines, I suppose we could have setup some backup CLI or AWS CodePipeline (eww) workflows.

Re: GitHub Git Operations Are Down

#254

Post by cyanydeez »

zuppy wrote: post #55
cyanydeez wrote: post #42

Earlier quoted context omitted.

Thats mostly because of herdmentality not because of a considered approach. Decentralization assumes the majority are anti social.

what a convoluted way to say you do not agree with this. when you are dealing with people of all skill levels, every step that you put on the development process costs real money. over that, not everything is and has to be open source. we all have source code that is proprietary and has a value for us as it is. making that decentralized and private is not something easy to achieve (and I don't really see a benefit).…

...ironic

Re: GitHub Git Operations Are Down

#255

Post by sashank_1509 »

shaftway wrote: post #234
sashank_1509 wrote: post #204

Earlier quoted context omitted.

I for a second thought was I getting fired

As someone who found out that way twice in the last two years, I also thought I had been let go. Anxiety is a hell of a drug.

Sorry for you, it should be illegal for companies to fire with no notice!

Re: GitHub Git Operations Are Down

#256

Post by bastardoperator »

vv_ wrote: post #223
liontwist wrote: post #125

Earlier quoted context omitted.

They are advocating for decentralized tools. That’s actionable and hoping to prevent failure.

There is no reason to host 'decentralized' tools besides regulation. It's considerably cheaper to use GitHub (or other alternatives like GitLab) than hosting your own and hiring people to maintain and support the solution. Their issue tracking system is very convenient for small teams too.

Exactly, hire a team of 3 and pay 500K in compensation, or spend 100K on a system that works and you get a support person to call in the event of an issue. The math is so simple.

Re: GitHub Git Operations Are Down

#257

Post by liontwist »

vv_ wrote: post #223
liontwist wrote: post #125

Earlier quoted context omitted.

They are advocating for decentralized tools. That’s actionable and hoping to prevent failure.

There is no reason to host 'decentralized' tools besides regulation. It's considerably cheaper to use GitHub (or other alternatives like GitLab) than hosting your own and hiring people to maintain and support the solution. Their issue tracking system is very convenient for small teams too.

> It's considerably cheaper to us

This is not true. The cheapest option is to not have services that require servers to maintain. Git continues to work if GitHub is down. So do shell scripts when CI is down. So why can’t we have an issue system where the underlying data is text files in a git branch?

I understand at scale you can pay people to optimize a process for the larger team, but there is a ton of unnecessary fragility before getting to that scale.

Re: GitHub Git Operations Are Down

#258

Post by liontwist »

bastardoperator wrote: post #256
vv_ wrote: post #223

Earlier quoted context omitted.

There is no reason to host 'decentralized' tools besides regulation. It's considerably cheaper to use GitHub (or other alternatives like GitLab) than hosting your own and hiring people to maintain and support the solution. Their issue tracking system is very convenient for small teams too.

Exactly, hire a team of 3 and pay 500K in compensation, or spend 100K on a system that works and you get a support person to call in the event of an issue. The math is so simple.

Except you’re not considering the cost of when you can’t deliver something on time for a customer because infrastructure you don’t control is down.

You don’t outsource things that prevent you from doing your core competency.

Re: GitHub Git Operations Are Down

#259

Post by bastardoperator »

liontwist wrote: post #258
bastardoperator wrote: post #256

Earlier quoted context omitted.

Exactly, hire a team of 3 and pay 500K in compensation, or spend 100K on a system that works and you get a support person to call in the event of an issue. The math is so simple.

Except you’re not considering the cost of when you can’t deliver something on time for a customer because infrastructure you don’t control is down. You don’t outsource things that prevent you from doing your core competency.

And for most companies, building and managing an SCM is absolutely not their core competency. Your point is valid, but not in the way you're trying to convey it.

Re: GitHub Git Operations Are Down

#260

Post by confooseddd »

Dylan16807 wrote: post #194
confooseddd wrote: post #165

Earlier quoted context omitted.

A CI infra just needs to communicate. It can communicate over email, like it does for the kernel. A bug tracker is just assorted communication. One can easily build it over email. You're just indulging in hyperbole for the sake of it. Nobody said git has an issue tracker in it.

> You're just indulging in hyperbole for the sake of it. Nobody said git has an issue tracker in it. Yes they did. That's what this comment thread is about. https://news.ycombinator.com/item?id=42691624 (Unless you're splitting some really fine hairs about what "in it" means?)

They just mentioned the most common communication tool used with Git? You're the one splitting hairs here.
Post reply on HN