Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

101–110 of 193 posts

Re: Update on 1/28 service outage

#101

Post by Domenic_S »

cookiecaper wrote: post #23
bhaak wrote: post #10

"Millions of people and businesses depend on GitHub" Well, we shouldn't depend on it so much. I shudder at the thought what an outage of GitHub would mean for our company. This time, we were lucky as it was during the night in Europe. Unfortunately, I don't have the power to test this scenario in our company.

It shouldn't really have much effect. One of git's major selling points is that it's a DVCS, meaning that everyone has a local copy of the repository. Perhaps some collaboration features will be down for a couple of hours (which I think is a downside to GitHub's decision not to put issues/PR history inside of git), but everyone should still be able to do work, commit to the repo, review history, and so forth. If you…

All sorts of other dependencies go down though. Packages you need for your build aren't there. CI or testing integrations don't happen. Code review is probably not happening. If you track issues in GH you can't see what's next to work on or look up requirements.

You're right in that you're (probably) not totally deadlocked. But I can't start to estimate the lost $$ in productivity that comes with a global GH outage because of all that.

Re: Update on 1/28 service outage

#102

Post by brinker »

viperscape wrote: post #50
rburhum wrote: post #37

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…

The package system for the rust language actually relies on github, as many found out during outage. I don't know if that will change, probably will with a read copy in a different git service.. but I thought it was interesting because I use github for everything save a few private projects, as I imagine most do. I'm not sure what to think of this, it seems backwards and grossly incompetent, yet here we are using it…

> The package system for the rust language actually relies on github, as many found out during outage.

This is not quite correct (although close to it). Cargo doesn't rely on GitHub, but it expects that there is some publicly-accessible git repository from which it can pull the source for any crate, and most crates use GitHub. So it's not a particular choice of Cargo, but a side-effect of GitHub's popularity in the community, and the fact that Cargo does not host source code itself.

Re: Update on 1/28 service outage

#103

Post by tehbeard »

nickpsecurity wrote: post #76
ajkjk wrote: post #69

Earlier quoted context omitted.

I'm sorry to be rude, but, it sounds like you should go learn Git and come back to this conversation. The decentralized design does avoid single points of failures, and everyone does have a copy. So - check, check, great. Unfortunately (maybe..) everyone has put their master repos in the same place, which somewhat counteracts the decentralization. But there is certainly no immediate coupling between the Git repositor…

"I'm sorry to be rude, but, it sounds like you should go learn Git and come back to this conversation." It looks like it. "The decentralized design does avoid single points of failures, and everyone does have a copy. " So, like many decentralized systems I've used, a master node gets worked around by other nodes who communicate in another way? Or would some retarded situation be possible where... "Unfortunately (mayb…

It's a pebkac issue. The software is fully capable of having multiple remotes, but it's rarely used that way.

Re: Update on 1/28 service outage

#104

Post by nickpsecurity »

kbenson wrote: post #91
nickpsecurity wrote: post #72

Earlier quoted context omitted.

Hence Git/Github in my comment. I already know there's a difference. I just don't know much more than that until I learn the two.

Github is to git as Sourceforge is (used to be) to subversion, but with a better UI. And yes, there have been concerns raised about what would happen if Github took a turn like Sourceforge, which usually get brought up when information about new shady practices at Sourceforge come up (or they get rehashed here).

Makes sense. I'm quite interested in seeing where it goes over time. I think it will depend a lot on the nature of the company. If it's VC-funded & aiming for acquisition, then there's a decent chance of Sourceforge history repeating. Otherwise, it might stick around as a beneficial ecosystem. Time will tell.

Re: Update on 1/28 service outage

#105

Post by nickpsecurity »

tehbeard wrote: post #103
nickpsecurity wrote: post #76

Earlier quoted context omitted.

"I'm sorry to be rude, but, it sounds like you should go learn Git and come back to this conversation." It looks like it. "The decentralized design does avoid single points of failures, and everyone does have a copy. " So, like many decentralized systems I've used, a master node gets worked around by other nodes who communicate in another way? Or would some retarded situation be possible where... "Unfortunately (mayb…

It's a pebkac issue. The software is fully capable of having multiple remotes, but it's rarely used that way.

Lol. Nicely put.

Re: Update on 1/28 service outage

#106

Post by jessaustin »

forrestthewoods wrote: post #42
rburhum wrote: post #37

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…

And people give me shit when I argue that open source projects should include 100% of dependencies.

If you personally run software for which reliability is important, absolutely you should maintain your own vendor repos. Open source projects are not in that position, and following your advice would lead to much harmful coupling and repetition.

Re: Update on 1/28 service outage

#107

Post by JetSetWilly »

drather19 wrote: post #73
JetSetWilly wrote: post #48

Earlier quoted context omitted.

The problem is most environments are very heteregenous. I evaluated chaos monkey approach for a big bank, the issue is that netflix has whole data centres full of loads of machines doing pretty much the same thing, streaming and serving. And the worst that can happen is a customer's stream stops and they have to restart it. But in most big companies you have thousands of apps that are all doing very different things.…

While Netflix as a company is focused at doing one specific thing at large scale, they're heavily vested in microservices and do actually have "thousands of apps that are all doing very different things". Chaos Monkey fits when people build and deploy their services with the notion that any particular instance (or dependency) could fail at any given time. It's a tough road to evolve out of a legacy, monolithic stack…

Whether they have broken up their apps into microservices doesn't seem to matter to me. That's just a matter of how they have organised their code, whether the actual app is monolithic or microlithic doesn't seem to matter.

They have a focussed business with relatively little variation in how they make money - all their customers simply pay for a streaming service.

Most large companies, certainly banks anyway, have thousands of apps because there's also thousands of different parts of the business making money in their own unique ways that have their own unique needs.

What works for netflix therefore can't work for other businesses, because the actual business is much more heterogenous than that of netflix and the technology will reflect that whether it is organised in microservices or monolithically - that's totally irrelevant.

Re: Update on 1/28 service outage

#108

Post by nickpsecurity »

jacquesm wrote: post #99
nickpsecurity wrote: post #96

Earlier quoted context omitted.

Haha nice catch. I missed it originally thinking it would just be more marketing crap. So, they probably just moved it before detonating. Not sadistic bastards after all.

Watch 'The Prestige'.

I know... My original reply mentioned two scenarios with one having replacement fish. Then, I thought people would think I'm overly paranoid or negative. I just couldn't help wonder if they'd blow the fish for fun then avoid liability with similar looking ones. Then, I edited the comment for sake of presumption of innocence.

But, yeah, I hear you... Great movie as well. One of few that brings my favorite mad scientist into eye of mainstream audience as well. I doubt I must name him. :)

EDIT to add: I'm guessing you think the geeks were too sadistic to pass up the opportunity, eh?

Re: Update on 1/28 service outage

#109

Post by Sanddancer »

pc86 wrote: post #31
raverbashing wrote: post #28

Earlier quoted context omitted.

Have a local repos that mirrors the master one on GitHub periodically Should that fail, start working on the local repos until github is back, then sync back to it

Depending on your definition of "periodically" you may lose almost as much time to syncing back than the outage would have caused without the local mirrors.

I've written scripts that do this. Any request for a repo is polled against the local repo server that makes sure it has the repo, and then quickly checks to see if the repo's out of date, caching the resulting file if the repo can be reached. If the repo can't be reached, just have the proxy deliver the old fileset. So the local repo gets updated, or at least attempts to update, with every hit against it. I had some other logic in the script to only check freshness every 10-15 minutes, so that during times when a lot of machines were pulling, they were essentially guaranteed to all get the same version.

Re: Update on 1/28 service outage

#110

Post by nickpsecurity »

jacquesm wrote: post #97
jakobegger wrote: post #85

Earlier quoted context omitted.

Watch it until the end :)

I did, but unless the whole video was a fake it doesn't really matter does it? And if it is then that does not reflect well on HP either...

I didn't think about the image angle. Yeah, you'd think a marketing person would be like, "Wait, this could lead to a PETA lawsuit and lower sales. Not to mention our segment that likes fish."
Post reply on HN