Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

151–160 of 193 posts

Re: Update on 1/28 service outage

#151

Post by gregmac »

Pharaoh2 wrote: post #147
Zelmor wrote: post #94

Earlier quoted context omitted.

One has to keep in mind that there is no cloud. It's just someone else's computer.

Except that it is not. It redundant array of computers, if one goes down, another takes it place and all the apps running on it are migrated to the new hardware. And if the whole zone goes down, the apps are migrated to a different zone. If the whole region goes down, the apps can be migrated to a different region. The 9s are so high that you don't have to worry about hardware issues anymore, unlike when you are runn…

Or it could be literally an old desktop computer sitting in someone's damp basement on a DSL connection. The problem with just saying "the cloud" is you can't tell the difference.

Re: Update on 1/28 service outage

#152

Post by tylorr »

chatmasta wrote: post #124
rburhum wrote: post #37

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…

If this is a problem worth solving, you can absolutely solve it. The easiest would be through the use of a caching proxy and/or load balancing system. The caching proxy system could be as simple as setting up a squid cache for apt. Multiple projects exist which do this already. The load balancing system would involve keeping a private mirror of every repository in the dependency graph, and falling back to the mirror…

For the load balancing system I would assume you would also want to keep the mirrors up to date as well? So for every request if mirror exists and is out of date, update it.

Re: Update on 1/28 service outage

#153

Post by chatmasta »

tylorr wrote: post #152
chatmasta wrote: post #124

Earlier quoted context omitted.

If this is a problem worth solving, you can absolutely solve it. The easiest would be through the use of a caching proxy and/or load balancing system. The caching proxy system could be as simple as setting up a squid cache for apt. Multiple projects exist which do this already. The load balancing system would involve keeping a private mirror of every repository in the dependency graph, and falling back to the mirror…

For the load balancing system I would assume you would also want to keep the mirrors up to date as well? So for every request if mirror exists and is out of date, update it.

Presumably that would be the responsibility of the mirroring system once a mirror is initiated.

Re: Update on 1/28 service outage

#154

Post by brazzledazzle »

tommoor wrote: post #6

This post makes it sound like Github has it's own data centers and power infrastructure structure, this is definitely news to me.. I'd presumed co-lo at best.

Unless it's explicitly stated to be a wholly owned data center I always assume companies are talking about rack space in a bigger DC like supernap.

Re: Update on 1/28 service outage

#155

Post by matt_wulfeck »

Why is it so hard for us to distribute our dependencies? Hash the package to a sha and put t anywhere on the Internet. Then we just need a service that holds and updates the locations of the hashes and we can fetch them anywhere.

Re: Update on 1/28 service outage

#156

Post by cdelsolar »

forrestthewoods wrote: post #149
kbenson wrote: post #55

Earlier quoted context omitted.

I think that's a bit crazy as well. This is a problem if your build process happens often and requires pulling external data. Ideally, you want a way to cache that external data, and a way to force invalidation of that cache. Building , at least after the first time, should not require external access. There are security reasons for this as well.

So your proposed solution is one of the only two hard problems in computer science? That should be a solid clue that you're wrong. "There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton

wow, i never realized cache invalidation was one of the ONLY two hard problems in CS

Re: Update on 1/28 service outage

#157

Post by rcthompson »

ceejayoz wrote: post #20
frik wrote: post #3

You can see the cascade effect on their status page graphs: https://status.github.com/

Interesting that their exception logging didn't get turned back on until this morning, from the looks of things.

Well, if exception logger was going off nonstop due to the outage, yet not providing any new information, it would make sense if they disabled it until things had returned to normal.

Re: Update on 1/28 service outage

#158

Post by semi-extrinsic »

Zelmor wrote: post #94
hguant wrote: post #86

Earlier quoted context omitted.

I think a large part of this is that cloud hosting has allowed us to abstract those problems - reliability, disaster recovery, geographical distribution - away, and we don't really think of computers as computers anymore. It's a service or a platform or what have you, and the expectation is that it will always be there. I wouldn't say this is laziness, just a byproduct of changing how we view Internet architecture. W…

One has to keep in mind that there is no cloud. It's just someone else's computer.

https://xkcd.com/908/

Re: Update on 1/28 service outage

#159

Post by ugexe »

swalsh wrote: post #141
rburhum wrote: post #37

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…

I mean a really simple solution (simple to say, maybe not to do) would be for package managers to require a "backup" repository from a different domain, than if you get a 500 error try the second remote repository. Use git for its advantages.

I think you mean a mirror, and many package managers use them.

Re: Update on 1/28 service outage

#160

Post by debaserab2 »

nickpsecurity wrote: post #72
debaserab2 wrote: post #65

Earlier quoted context omitted.

git != github

Hence Git/Github in my comment. I already know there's a difference. I just don't know much more than that until I learn the two.

If you understand the difference between the two, you'd realize your comment makes no sense. The fact that github went down due to a power failure has nothing to do with git as a solution.

The fact that everyone uses git more or less the same as svn is the problem. Git is decentralized, but because so many people rely on github most don't ever use the decentralized aspect to it.

Post reply on HN