Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

131–140 of 193 posts

Re: Update on 1/28 service outage

#131

Post by maker1138 »

nickpsecurity wrote: post #59
rburhum wrote: post #37

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…

That's a good point. I've been ignoring learning Git as long as I can but almost everything on my todo list heavily uses it. Or ties into it as you said. So, I'm going to have to bite the bullet and learn it. Yet, I swore Git fans told me its decentralized design avoids single points of failures where everyone has a copy and can still work when a node is down just not necessarily coordinate or sync in a straight-forw…

Git is to GitHub as JavaScript is to Java. Though their names are similar they are very different things.

Re: Update on 1/28 service outage

#132

Post by stouset »

brinker wrote: post #102
viperscape wrote: post #50

Earlier quoted context omitted.

The package system for the rust language actually relies on github, as many found out during outage. I don't know if that will change, probably will with a read copy in a different git service.. but I thought it was interesting because I use github for everything save a few private projects, as I imagine most do. I'm not sure what to think of this, it seems backwards and grossly incompetent, yet here we are using it…

> The package system for the rust language actually relies on github, as many found out during outage. This is not quite correct (although close to it). Cargo doesn't rely on GitHub, but it expects that there is some publicly-accessible git repository from which it can pull the source for any crate, and most crates use GitHub. So it's not a particular choice of Cargo, but a side-effect of GitHub's popularity in the c…

If crates.io hosted the content themselves, it would just be the same problem, only with a service potentially less reliable than GitHub.

Re: Update on 1/28 service outage

#133

Post by why-el »

paultela wrote: post #77
why-el wrote: post #44

Earlier quoted context omitted.

> if you're not regularly cutting power to your data center, you're not building resilience to such a thing happening Would love to read examples on who is doing this and how? Reminds me of Netflix's Choas monkey, only applied to electricity. :p

There's a mention of Facebook regularly doing this in the summary section of this instagram engineering post: http://engineering.instagram.com/posts/548723638608102/ EDIT: Here's more info: http://www.datacenterknowledge.com/archives/2014/09/15/faceb...

Awesome, thank you. :)

Re: Update on 1/28 service outage

#134

Post by JonathonW »

zwetan wrote: post #115
kbenson wrote: post #55

Earlier quoted context omitted.

I think that's a bit crazy as well. This is a problem if your build process happens often and requires pulling external data. Ideally, you want a way to cache that external data, and a way to force invalidation of that cache. Building , at least after the first time, should not require external access. There are security reasons for this as well.

It's not, if you can not even run that first build then you actually have nothing to work on. Also, not frozen dependencies means you are at the mercy on any dependencies changes breaking your build at any time. With that, even if your first build run and go fetch those deps and can build at T1, it is not guaranteed at all that the build will work at T1+n. There is a big difference between your team working from trun…

Just because you're downloading your dependencies at runtime doesn't mean you have to have non-frozen dependencies or non-repeatable builds... that's one of the advantages of pulling dependencies out of a Git repository; specify a specific revision to build against and that code is guaranteed* to not change. Pulling dependencies from Git doesn't mean you're working against trunk.

Now, if you're doing this with mission-critical software, you should probably be maintaining mirrors of those dependencies locally on infrastructure you control, but, again, that's another of the things that Git makes easy.

You should never be dependent on a reference that can move, unless you're willing to accept the consequences (that includes branches in any version control system, tags if you don't have infrastructure to verify that they haven't changed, external non-version-controlled downloads, etc.).

Basically, what you should learn here is that you shouldn't build your business around a third-party service's continued availability. Especially if it's a third-party service where you're not paying for an SLA, like Github. Reproducibility of builds is a different issue, and including 100% of your dependencies in your own source repository is not the only solution to it.

* Barring a SHA-1 collision, which is highly unlikely with Git.

Re: Update on 1/28 service outage

#135

Post by kbenson »

stouset wrote: post #132
brinker wrote: post #102

Earlier quoted context omitted.

> The package system for the rust language actually relies on github, as many found out during outage. This is not quite correct (although close to it). Cargo doesn't rely on GitHub, but it expects that there is some publicly-accessible git repository from which it can pull the source for any crate, and most crates use GitHub. So it's not a particular choice of Cargo, but a side-effect of GitHub's popularity in the c…

If crates.io hosted the content themselves, it would just be the same problem, only with a service potentially less reliable than GitHub.

Unless you set up a system of mirrors. There are plenty of examples[1] to draw from.

1: http://mirrors.cpan.org/

Re: Update on 1/28 service outage

#136

Post by nickpsecurity »

nemothekid wrote: post #130
nickpsecurity wrote: post #119

Earlier quoted context omitted.

Thanks for the extra detail. "and there is no way to configure mirrors." Its that in Git itself or the project-specific tooling you're mentioning?

There is no way to configure mirrors with the project-specific tooling (AFAIK). Git, (and like most other DVCS) supports mirroring. For example Linux, hosted on Github, ( https://github.com/torvalds/linux/commits/master ) is also mirrored and hosted on kernel.org ( https://git.kernel.org/cgit/ ). Or, the apache projects ( https://github.com/apache/cassandra ), which are also hosted on apache.org ( https://git-wip-us.…

So, the protocol is definitely good enough to handle situations like this but just commonly applied that way esp with many Github-hosted projects. Gotcha. That makes sense.

Re: Update on 1/28 service outage

#137

Post by st3v3r »

stouset wrote: post #132
brinker wrote: post #102

Earlier quoted context omitted.

> The package system for the rust language actually relies on github, as many found out during outage. This is not quite correct (although close to it). Cargo doesn't rely on GitHub, but it expects that there is some publicly-accessible git repository from which it can pull the source for any crate, and most crates use GitHub. So it's not a particular choice of Cargo, but a side-effect of GitHub's popularity in the c…

If crates.io hosted the content themselves, it would just be the same problem, only with a service potentially less reliable than GitHub.

Not if they acted like a mirror. Put it on GitHub, Bitbucket, and crates.io

Re: Update on 1/28 service outage

#138

Post by nickpsecurity »

jallmann wrote: post #129
nickpsecurity wrote: post #122

Earlier quoted context omitted.

Makes sense. I'll try to remember that for my future checklist. Thanks for the details. Btw, you're site is down on my end from 2 browsers on my desktop and one on mobile. Might want to look into that as rest are working.

> Btw, you're site is down on my end Hah, because it's been defunct for a while now. Thanks for the reminder, removed it from my profile.

Cool

Re: Update on 1/28 service outage

#139

Post by sre_ops »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

Welcome to reality. Power outages create real problems and they rarely go as planned.

Re: Update on 1/28 service outage

#140

Post by cesarb »

nemothekid wrote: post #130
nickpsecurity wrote: post #119

Earlier quoted context omitted.

Thanks for the extra detail. "and there is no way to configure mirrors." Its that in Git itself or the project-specific tooling you're mentioning?

There is no way to configure mirrors with the project-specific tooling (AFAIK). Git, (and like most other DVCS) supports mirroring. For example Linux, hosted on Github, ( https://github.com/torvalds/linux/commits/master ) is also mirrored and hosted on kernel.org ( https://git.kernel.org/cgit/ ). Or, the apache projects ( https://github.com/apache/cassandra ), which are also hosted on apache.org ( https://git-wip-us.…

> For example Linux, hosted on Github, (https://github.com/torvalds/linux/commits/master) is also mirrored and hosted on kernel.org (https://git.kernel.org/cgit/).

It's the opposite: Linux is hosted on kernel.org, and the mirror on github.com is just something that was created during a kernel.org outage. The canonical address is the kernel.org one.

(The Linux repository on kernel.org, by the way, is one of the oldest git repositories; IIRC, it was created when git was only a few weeks old.)

Post reply on HN