Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: October 21 Incident Report

blog.github.com

Post reply on HN
Print view View original

111–119 of 119 posts

Re: GitHub: October 21 Incident Report

#111

Post by bschwindHN »

Klathmon wrote: post #53
_verandaguy wrote: post #49

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks b…

It's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's…

> and is probably more likely to go down.

People often say this but I wonder if it's true. Your replication wouldn't be handling the same load as GitHub itself, so perhaps the issues that GitHub experiences would rarely happen to a self hosted version.

Re: GitHub: October 21 Incident Report

#112

Post by nineteen999 »

_verandaguy wrote: post #49

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks b…

This is why we built and manage our own git servers internally. We run emergency services infrastructure so we can not afford to be disconnected from our version control. Ever.

Same reasons we don't run our application and supporting infrastructure in a cloud provider really - apart from the complexity of our particular network which has a heavy dependency on mobile and satellite links - we cannot guarantee our customer of our availability if we don't control the compute, storage, SAN fabric, and as much of the network infrastructure as is possible.

Re: GitHub: October 21 Incident Report

#113

Post by platinium »

goykasi wrote: post #80
platinium wrote: post #71

Earlier quoted context omitted.

Going forward, I see that the sensible solution for all small companies relying on cloud-hosted git is to always have a secondary cloud provider at all times.

> Which brings into question... what is so special about the "front-end" features that Git provides? ... bare bones skeleton that we can swap in/out at will Your comment disappeared. I think you had a good point. And itd be cool if it could be a real thing, but... I dont disagree. But our team realized that we rely almost too much on github; we just decided to put up with it. Is there a solution that doesnt depend on…

Ah yeah, it disappeared because I thought I would get more reads If I moved it up in the post chain lol.

May I ask what makes your company feel secure about using cloud hosted solutions. I mean, can’t a disgruntled employee easily clone the git repo to his own GitHub account ? I suppose they could do that anyways, just by copying the repo somewhere, but having an entire company’s secret code on the cloud just seems to remove too many barriers of entry for protecting the code.

Re: GitHub: October 21 Incident Report

#114

Post by alexeiz »

goykasi wrote: post #69
alexeiz wrote: post #45

Earlier quoted context omitted.

The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project d…

But you are ignoring a huge component of using github. If you only use github as a central, shared repo, sure you lost almost nothing. Their git infra seemed to be operational throughout this, but if you use the major features of github: issues, PRs, review, webhooks for CI, etc, you probably did lose out on developer time. Me and my team pushed tons of code to origin today (JST btw), but we were almost at a stand st…

Almost everything you mentioned can be done without Github. Add a new remote, make teammates push branches to that remote. Merge to master locally. Review old-school way with 'git diff'. Apart from issues and CI, progress can still be made if you use Git features that make it distributed in the first place. If you depend on Github that much, you should think about a fallback strategy. Otherwise your business model is just unsound.

Re: GitHub: October 21 Incident Report

#116

Post by h1d »

_verandaguy wrote: post #49

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks b…

Why would state actor level attacker target for being a code hosing platform? At best they can temporarily suspend operations.

Re: GitHub: October 21 Incident Report

#117

Post by h1d »

elliotlarson wrote: post #57
colonelpopcorn wrote: post #56

Earlier quoted context omitted.

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.

I'm not sure how hard it is to maintain some self hosted apps. It's just set up and forget. It doesn't randomly change interface or license unless you upgrade it as well.

I've used AWS for 10 years but for last 5 years, I've never seen it just go down randomly and even if it did, you have the room to redeploy with some clicks (assuming you have your data backed up regularly) instead of waiting for the uncontrollable.

You seem to take the ops work a bit overly.

Re: GitHub: October 21 Incident Report

#118

Post by caspar »

nik736 wrote: post #6

I don't understand why code hosting platforms like GitHub, GitLab or BitBucket have so many issues regularly. Is there anything special about it?

What's special about it is that you have terabytes of data to keep available, and at scale git does not play well with technologies like NFS or cloud object storage like S3, so each major provider either pays a lot of money to specialized vendors or has homegrown solutions to deal with the problem.

So on top of your usual problems with keeping a cloud service up and running, you also have that git IO problem to contend with, and to rub salt in the wound, that wrinkle also makes it difficult to fully adopt many "standard" cloud architectures or vendors (such as AWS) which work for non-IO-heavy applications: you always have this major part of your infrastructure that has this special requirement holding you back at least partially (and that can hurt your availability for related services which are not even IO-heavy).

(That said, it's hard to guess whether that was the problem, a contributing factor, or unrelated entirely based on the details provided here.)

source: I work at Atlassian (though not on the Bitbucket team) and occasionally chat to current and former Bitbucket devs on this topic.

Re: GitHub: October 21 Incident Report

#119

Post by lugg »

metaphor wrote: post #50
lugg wrote: post #30

Earlier quoted context omitted.

> Define regularly More often than what is considered a standard 99.99% uptime SLA? (about an hour per year.) You seem to be making it out like a couple of days a year of lost [1] developer productivity is no big deal. That said, these things happen and you should probably check your workflows if you're all that blocked by GitHub being down.

> More often than what is considered a standard 99.99% uptime SLA? GitHub SLA is 99.95% and apparently exclusive to Business Cloud customers[1]. [1] https://github.com/pricing

I wasn't saying it applied. Just what's expected from a large international company relied on by so many.
Post reply on HN