The package system for the rust language actually relies on github, as many found out during outage. I don't know if that will change, probably will with a read copy in a different git service.. but I thought it was interesting because I use github for everything save a few private projects, as I imagine most do. I'm not sure what to think of this, it seems backwards and grossly incompetent, yet here we are using it…
Not just rust language, to the best of my knowledge, even packagist, the php package manager relies heavily on github for sourcing its packages. But I think they have other resources too, apart from github.
Ruby's bundler doesn't entirely rely on Github, but pulling from a Github repo is a supported option that many take advantage of.
Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…
And people give me shit when I argue that open source projects should include 100% of dependencies.
how far down the stack do you go? do open source projects need to include their own compiler? what would compile it?
Im not positive, but it sounds like a fairly recent switch from a cloud provider to their own datacenter. If thats the case, Id expect a number of outages to come in the following months.
I'm starting to think that people should mirror their packages to BitBucket as a rule, and that package managers should round robin/flip a coin between the two, or use whichever is available in case of outages.
I'd rather have something like Netflix's Open Connect Appliances, covering all of Github, sitting in each office and a centrally located colo facility.
I'm not familiar with Open Connect Appliances, but Github as a platform is still a Single Point of Failure at least on some level. Domain or SSL issues, for example.
I think that as long as we have options to host packages on other platforms in addition, it should be seriously considered. At the very least, it would encourage a more competitive atmosphere for open source hosting services.
We've been mitigating against this kind of thing with backups at other datacenters or colos for a while. They can be hot standby, cold standby, slightly degraded in performance, whatever. I also recommend the backup be on a different part of the overall power grid in case it cascades in failure. The good colo's often have connections to multiple backbones, too, which is extra redundancy. That all assumes there's a to…
I doubt it matters to anybody but was it really necessary to kill the fish?
Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…
if you dig far enough, you can find some dependency of a dependency of dependency that relies on GitHub. Brew/npm/build script/whatever But really, why? Is it just institutional laziness on the part of all developers? We had reliable rsync CPAN mirrors in 1995. In the early days of the Internet, companies would mutually host secondary DNS for each other to be more reliable. For some reason, we've forgotten all about…
I think a large part of this is that cloud hosting has allowed us to abstract those problems - reliability, disaster recovery, geographical distribution - away, and we don't really think of computers as computers anymore. It's a service or a platform or what have you, and the expectation is that it will always be there. I wouldn't say this is laziness, just a byproduct of changing how we view Internet architecture. We systems to take care of reliability etc because everyone has those problems. Now, those are only things you'll experience if you host your own stuff, or work for one of the big providers. (Broad assertion, I know, but I think it's mostly true)
"Millions of people and businesses depend on GitHub" Well, we shouldn't depend on it so much. I shudder at the thought what an outage of GitHub would mean for our company. This time, we were lucky as it was during the night in Europe. Unfortunately, I don't have the power to test this scenario in our company.
I like others am confused by this common sentiment. Github is the remote repo, but the version control is distributed so everyone has a copy. I'm pretty sure I can fill a few hours or more with work needing to be done on my local repo. FYI I'm not a professional software developer but I would like to know. The things that come to mind: issue trackers, messaging, not being able to see latest pull requests. Update: Now…
Not much detail here. A more thorough postmortem would give me more confidence they can recover from another similar issue. Hoping to see one soon.
Yep, I think most of these post-mortems from any company are pointless from a technical perspective. It's 4 paragraphs that boils down to "someone did something wrong and we'll make sure it doesn't happen" with zero specifics. There's no point in reading these because there's no technical information. Stuff like this is something you sent to your customer because they want root cause.
I strongly disagree that these sorts of communications are pointless. In every major service outage I've seen where the company maintained a degree of silence, it's caused major damage to their public relations and consumer trust.
I know it doesn't tell you much about exactly what happened, but the truth is they may still be sorting that out and focusing on ensuring it does not happen again. An in-depth post-mortem accompanied by a description of the fix would be great. In the meantime, admitting culpability and apologizing are the ideal essential first steps.
I've experienced a brief full-scale power loss at a data center before. It is unbelievable how much goes wrong. The machines had been chugging along for years, happily doing their job, but on the next boot the hard drives were suddenly corrupted, or the power supplies broken. The impacts of that power outage were felt for at least six months. It's one of those things where, if you're not regularly cutting power to yo…
> if you're not regularly cutting power to your data center, you're not building resilience to such a thing happening Would love to read examples on who is doing this and how? Reminds me of Netflix's Choas monkey, only applied to electricity. :p
I remember reading a few years back that Yahoo once a week takes a random data center offline, just to make sure they could do that without issues. They probably didn't actually cut the power ;) But they used it as an argument against investing to much in emergency generators and such: they'll fail or cause accidents and you need the ability to fail-over either way, so make it routine.
This looks like a single datacenter. I don't see anything here indicating high availability or other datacenters. You'll usually spot either an outright mention of it or certain components/setups common in it. They might have updated their stuff for redundancy since then. However, if it's same architecture, then the reason for the downtime might be intentional design where only a single datacenter has to go down.
Might be fine given how people apparently use the service. It's just good to know that this is the case so users can factor that into how they use the product and have a way of working around the expected downtime if it's critical to do so.