Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

121–130 of 193 posts

Re: Update on 1/28 service outage

#121

Post by pc86 »

Sanddancer wrote: post #109
pc86 wrote: post #31

Earlier quoted context omitted.

Depending on your definition of "periodically" you may lose almost as much time to syncing back than the outage would have caused without the local mirrors.

I've written scripts that do this. Any request for a repo is polled against the local repo server that makes sure it has the repo, and then quickly checks to see if the repo's out of date, caching the resulting file if the repo can be reached. If the repo can't be reached, just have the proxy deliver the old fileset. So the local repo gets updated, or at least attempts to update, with every hit against it. I had some…

This is certainly one of the better ways to do it - when I see a word like periodically I assume it means daily/weekly/on some sort of calendar-based schedule, which isn't necessarily the case here.

Re: Update on 1/28 service outage

#122

Post by nickpsecurity »

jallmann wrote: post #118
nickpsecurity wrote: post #59

Earlier quoted context omitted.

That's a good point. I've been ignoring learning Git as long as I can but almost everything on my todo list heavily uses it. Or ties into it as you said. So, I'm going to have to bite the bullet and learn it. Yet, I swore Git fans told me its decentralized design avoids single points of failures where everyone has a copy and can still work when a node is down just not necessarily coordinate or sync in a straight-forw…

> either for Git or just Github, there's some gap between the ideal they described and how things work in practice The hub-spoke topology is the easiest way of distributing source code to a lot of people. If the hub goes down, this is what happens. If that leads to a halt in productivity, then that is a failure in contingency planning. Git gives you many tools to distribute your workflow, but that won't save you if y…

Makes sense. I'll try to remember that for my future checklist. Thanks for the details. Btw, you're site is down on my end from 2 browsers on my desktop and one on mobile. Might want to look into that as rest are working.

Re: Update on 1/28 service outage

#123

Post by nickpsecurity »

jacquesm wrote: post #112
nickpsecurity wrote: post #108

Earlier quoted context omitted.

I know... My original reply mentioned two scenarios with one having replacement fish. Then, I thought people would think I'm overly paranoid or negative. I just couldn't help wonder if they'd blow the fish for fun then avoid liability with similar looking ones. Then, I edited the comment for sake of presumption of innocence. But, yeah, I hear you... Great movie as well. One of few that brings my favorite mad scientis…

> I'm guessing you think the geeks were too sadistic to pass up the opportunity, eh? No, it just seems weird, that's all. I don't see how either interpretation would benefit HP.

Oh OK. Makes sense.

Re: Update on 1/28 service outage

#124

Post by chatmasta »

rburhum wrote: post #37

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…

If this is a problem worth solving, you can absolutely solve it. The easiest would be through the use of a caching proxy and/or load balancing system.

The caching proxy system could be as simple as setting up a squid cache for apt. Multiple projects exist which do this already.

The load balancing system would involve keeping a private mirror of every repository in the dependency graph, and falling back to the mirror when GitHub fails. To automate this, proxy all git requests. If github is up, let the request pass through. If no mirror exists for the repository, create one. If GitHub fails, fall back to the mirror.

Re: Update on 1/28 service outage

#125

Post by kbenson »

zwetan wrote: post #115
kbenson wrote: post #55

Earlier quoted context omitted.

I think that's a bit crazy as well. This is a problem if your build process happens often and requires pulling external data. Ideally, you want a way to cache that external data, and a way to force invalidation of that cache. Building , at least after the first time, should not require external access. There are security reasons for this as well.

It's not, if you can not even run that first build then you actually have nothing to work on. Also, not frozen dependencies means you are at the mercy on any dependencies changes breaking your build at any time. With that, even if your first build run and go fetch those deps and can build at T1, it is not guaranteed at all that the build will work at T1+n. There is a big difference between your team working from trun…

> It's not, if you can not even run that first build then you actually have nothing to work on.

Obviously you can run the first build. You wouldn't be using Github if you never got it working in the first place.

To clarify, setting up the build environment may require network access, but if the process of building requires it, there are many places where it can go wrong, both operationally and security wise.

> Also, not frozen dependencies means you are at the mercy on any dependencies changes breaking your build at any time. ...

I agree, but that's a separate discussion and doesn't really apply here. There's nothing preventing the pulling of a specifically tagged version for builds. If someone's build process that used Git for dependencies is not doing this, whether they are using Github or some internal server is irrelevant, the same problems apply.

Re: Update on 1/28 service outage

#126

Post by nickpsecurity »

ymse wrote: post #111
nickpsecurity wrote: post #59

Earlier quoted context omitted.

That's a good point. I've been ignoring learning Git as long as I can but almost everything on my todo list heavily uses it. Or ties into it as you said. So, I'm going to have to bite the bullet and learn it. Yet, I swore Git fans told me its decentralized design avoids single points of failures where everyone has a copy and can still work when a node is down just not necessarily coordinate or sync in a straight-forw…

> I used to also use append-only storage for changes in potentially buggy or malicious services. Sounds like that might be a good idea, too, to prevent some issues. In a certain sense, git is "append-only". If you change a commit in history, every ancestor commit will have its SHA hash changed. Naturally this will conflict with other copies of the repository. For backups you should do a "git clone --bare" which check…

I figure it's append only at protocol level. Usually a smart idea for SCM. Is that still true when the whole datacenter goes down in mid-operation? Typically varies from implementation to implementation of the concept.

Re: Update on 1/28 service outage

#127

Post by Symbiote »

zwp wrote: post #117
tehbeard wrote: post #103

Earlier quoted context omitted.

It's a pebkac issue. The software is fully capable of having multiple remotes, but it's rarely used that way.

Is there an easy config for that? Suppose I want to push to eg github and bitbucket (without sharing my creds with ifttt or similar)? Is a post-receive hook on a local pseudo-master the way to go?

See, for example, here: http://stackoverflow.com/questions/14290113/git-pushing-code...

    git remote set-url --add --push origin git://original/repo.git
    git remote set-url --add --push origin git://another/repo.git

Re: Update on 1/28 service outage

#128

Post by zwetan »

bhaak wrote: post #10

"Millions of people and businesses depend on GitHub" Well, we shouldn't depend on it so much. I shudder at the thought what an outage of GitHub would mean for our company. This time, we were lucky as it was during the night in Europe. Unfortunately, I don't have the power to test this scenario in our company.

The key word is infrastructure.

Github should not be the master, it should be a mirror of a company master that they host on their own server.

The main problem with that is some company do not want the cost of the infra + the cost of the sysadmin to set that up, etc.

The second problem is the build, even if you host your own repos, if all your dependencies are on github and you don't include them in the repo, then you are bust.

Re: Update on 1/28 service outage

#129

Post by jallmann »

nickpsecurity wrote: post #122
jallmann wrote: post #118

Earlier quoted context omitted.

> either for Git or just Github, there's some gap between the ideal they described and how things work in practice The hub-spoke topology is the easiest way of distributing source code to a lot of people. If the hub goes down, this is what happens. If that leads to a halt in productivity, then that is a failure in contingency planning. Git gives you many tools to distribute your workflow, but that won't save you if y…

Makes sense. I'll try to remember that for my future checklist. Thanks for the details. Btw, you're site is down on my end from 2 browsers on my desktop and one on mobile. Might want to look into that as rest are working.

> Btw, you're site is down on my end

Hah, because it's been defunct for a while now. Thanks for the reminder, removed it from my profile.

Re: Update on 1/28 service outage

#130

Post by nemothekid »

nickpsecurity wrote: post #119
nemothekid wrote: post #114

Earlier quoted context omitted.

> So, countering this risk is apparently a manual thing if one uses off-the-shelf tooling for Git. Not so much off-the-shelf tooling for Git, its more off-the-shelf tooling for Node/Ruby/Go/Rust/PHP. Nothing about Node's npm really requires it to depend on a single GitHub, in fact I think you can use any Git repo. Its just that most tend to use a single Git repo, and there is no way to configure mirrors.

Thanks for the extra detail. "and there is no way to configure mirrors." Its that in Git itself or the project-specific tooling you're mentioning?

There is no way to configure mirrors with the project-specific tooling (AFAIK).

Git, (and like most other DVCS) supports mirroring. For example Linux, hosted on Github, (https://github.com/torvalds/linux/commits/master) is also mirrored and hosted on kernel.org (https://git.kernel.org/cgit/). Or, the apache projects (https://github.com/apache/cassandra), which are also hosted on apache.org (https://git-wip-us.apache.org/repos/asf?p=cassandra.git). Generally when commits are merged with upstream, they are mirrored to all other hosts.

The tools, however, are generally configured with only the GitHub address (or the author of the tool only publishes to GitHub), and the tools (unlike say Perl's CPAN) don't offer to maintain mirrors of the libraries published. So when github is down, a tool like npm will give up, even though the author could have another git repo host elsewhere.

Post reply on HN