Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

status.github.com

Post reply on HN
Print view View original

231–240 of 253 posts

Re: GitHub was down

#231

Post by peterwwillis »

cjbprime wrote: post #132
0xbear wrote: post #116

Earlier quoted context omitted.

I'm not "missing" anything. I worked at Google for 7 years much of which was spent working on, you guessed it, distributed systems infrastructure. You guard against this by carefully canarying things and putting robust testing, monitoring, and deployment procedures in place. A release might take a few days, but you can be reasonably certain your users won't be your guinea pigs, and if shit does hit the fan, rollback…

Why are you continuing to assume that this outage was caused by a release of some kind?

Every change is a release if you squint right.

Re: GitHub was down

#232

Post by theshrike79 »

hk__2 wrote: post #198
curun1r wrote: post #186

Earlier quoted context omitted.

> What more should they do? How about storing issues and PRs in the actual git repository? They should index them for the UI, sure, but the source of truth should be a branch of the git repo just like it is with gh-pages. It should be possible to file a new issue by committing a markdown file to the correct branch and pushing it to Github. Their hub command line tool and their own client could facilitate adding all t…

> How about storing issues and PRs in the actual git repository? Access to the git repository is regulated; issues and PRs aren’t. Anybody can fill an issue/PR on your repo, but only you (and your team) can modify the repo. You’d need to also store all the comments on all issues and PRs, even closed/rejected ones. In some repositories, that’d be huge.

Of course the issue data in the repo should be considered read only.

And the amount of actual data needed to store issues is peanuts anyway.

Re: GitHub was down

#233

Post by 0xbear »

notyourday wrote: post #228
0xbear wrote: post #177

Earlier quoted context omitted.

It's called continuous delivery, man. Merge to master and it goes out the door automatically, even if it's busted to hell. All the cool kids are doing it.

How was it merged to master if it did not pass the tests?

Shitty tests? Lack of integration tests? Lack of test coverage for a particular scenario? Test environment that does not represent the config that's deployed in production? There are literally hundreds of reasons why things like that could happen. Depending on the system in question you could have an isolated environment on which you replay prod traffic before cutting a new release and then investigate failures. All new features are engineered so that they could be easily turned off using flags. Once that's done you could canary your release to, say, 5% of your user base (at Google it'd be in the least loaded Borg cell), and if something goes wrong, you quickly disable busted features in response to monitoring and alerts. You let that stew for a while, then deploy your release worldwide, and start working on the next one.

Re: GitHub was down

#234

Post by vertex-four »

lost_password wrote: post #224
vertex-four wrote: post #218

Earlier quoted context omitted.

Your comment is entirely nonsensical in this context. I want a way to be able to publish a repo and have the people subscribing to the repo be able to only pay attention to my changes in an automated way. This software currently doesn't implement that, as far as the guide that was linked suggests. It is therefore utterly useless - every time someone decides to grief my repo, it requires manual intervention to resolve…

you said > Is there, at least theoretically, a way to prevent other people from pushing to my repo? so I answered, _yes, theoretically_ we have ideas for how to implement that. you can also unfollow and block griefers, but so far pretty much everyone has been nice and we just havn't needed to implement that yet.

How do you intend to automatically resolve merge conflicts, which is what your document suggests you want to do?

Why is this not resolved by a good permissions model and the ability to fork? Why should my users have to care about blocking griefers when they just want to pull from repo?

Re: GitHub was down

#235

Post by justinclift »

javajosh wrote: post #137
tinalumfoil wrote: post #54

Earlier quoted context omitted.

> bored developers Bored? Git's a distributed version control system, so no excuses. Get back to work! But in all seriousness I kind of wish GitHub provided a way to mirror things like issues and PRs so you never have to be fully reliant on one service. Not being able to read these really does make it impossible to get work done offline.

It would be clever to store issues and PRs on a special branch in the repository. I can't currently think of why this wouldn't work (apart from business-wise, it would reduce lock-in considerably).

Well, it's unlikely to be feasible due to Git's on disk structure.

With git, every commit has an sha checksum of the commit contents & metadata. And each commit's metadata points to the previous commit.

So, if a maintainer wanted to fix a typo or do any other kind of correction/update to an old issue you'd need to rewrite every commit on disk since that one with the updated chain of checksums. eg from the fixed comment, to the head of the special issue/pr branch.

That alone would probably make syncing with the repo's a real pain for anyone working on medium to larger sized projects. ;)

Re: GitHub was down

#236

Post by syshum »

heartbreak wrote: post #230
syshum wrote: post #207

Earlier quoted context omitted.

And you believe that makes them an "Open Source Company" To me in order to be a Open Source company your primary product must itself be Open Source. RedHat as an example... RHEL is open source GitLab.. GitLab is open source These are open source companies Hiring a few devs for work on some side projects that are open source does not make one an Open Source Company If they open GitHub Core then they can claim to be a…

Aaron Patterson, who I'll continue to use as an example, works full time on Rails while being paid by Github. Rails is not a side project for him. Also as mentioned elsewhere in this thread, Github has published plenty of open source software. Electron, Atom, resque, and updates to git itself.

It is a side project for the company, as are all the other projects you mentioned

As I clearly stated, GitHub can not be a Open Source company simply because they hired some devs to work on Open Source

Do you consider Microsoft to be a Open Source Company?

Re: GitHub was down

#237

Post by ci5er »

kaikai wrote: post #27

I would love to see a chart of traffic to other sites when GitHub goes down. My bet is that HackerNews and Twitter both get significant spikes from all those bored developers.

Uuuh ... Shouldn't they be, you know, developing? Or has GitHub turned into Reddit when I wasn't looking?

Re: GitHub was down

#238

Post by kyberias »

0xbear wrote: post #177
kyberias wrote: post #140

Earlier quoted context omitted.

They don't do any testing before push to prod?

It's called continuous delivery, man. Merge to master and it goes out the door automatically, even if it's busted to hell. All the cool kids are doing it.

I know what it's called. So do they have tests or not?

Re: GitHub was down

#239

Post by bluehatbrit »

relaxitup wrote: post #161

Do these general Github outages affect GH Pages as well, or is that service portion segmented to some degree?

Pages are static sites served from separate infra from the main github.com from what I've heard, so you should be fine in almost all cases. I've never had one go down unless I pushed a dud build.

Re: GitHub was down

#240

Post by notyourday »

0xbear wrote: post #233
notyourday wrote: post #228

Earlier quoted context omitted.

How was it merged to master if it did not pass the tests?

Shitty tests? Lack of integration tests? Lack of test coverage for a particular scenario? Test environment that does not represent the config that's deployed in production? There are literally hundreds of reasons why things like that could happen. Depending on the system in question you could have an isolated environment on which you replay prod traffic before cutting a new release and then investigate failures. All…

Let me get it straight... So the advantage of CI/CD is not automatic testing + roll outs, rather it is removal of a test requirement? Why waste resources on CI/CD in that case? Just remove the test requirements and deploy. In fact, remove canary as well - the more traffic hits the broken release the faster it will become obvious that release is broken.

The above was sarcasm.

If your org has CD and it does not have CI validation step that provides 100% test coverage of your code, then you do not have a CD - you have MSP - Merge->Ship->Pray system in place.

Post reply on HN