Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

January 28th Incident Report

github.com

Post reply on HN
Print view View original

171–180 of 198 posts

Re: January 28th Incident Report

#171

Post by kuschku »

imbriaco wrote: post #81
pedalpete wrote: post #13

Does Github run anything like Netflix Simbian Army against it's services? As a company by engineers for engineers with the scale that github has reached, I'm a bit surprised they are lacking a bit more redundancy. Though they may not need the uptime of netflix, an outage of more than a few minutes on github could affect businesses that rely on the service.

Google "Netflix downtime" for evidence that Netflix also has outages. Google has outages, sometimes very significant ones of Google Apps. Facebook has outages. Complex systems fail. Period. All the time. Things like the Simian Army are fantastic tools that help you identify a host of problems and remediate them in advance, but they cannot test every combinatorial possibility in a complex distributed system. At the en…

I remember in 2013 a full-day outage of Google.

Re: January 28th Incident Report

#172

Post by vacri »

drdrey wrote: post #98
theptip wrote: post #67

Earlier quoted context omitted.

It's shocking that they don't at least have a read replica of their system in another 'AZ'. That's cloud hosting 101, and being self-hosted isn't an excuse to skimp on this. If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.

For people reading this, AZ in this context would be Availability Zone

For people reading this, 'Availability Zone' in this context would be (AWS speak for) 'datacentre'. :)

Re: January 28th Incident Report

#173

Post by majewsky »

cognivore wrote: post #124

Um, work from your local cache for a few hours? It's that the one of the main reasons for git?

Not all processes that involve GitHub are development processes. I've seen automated deployments fail inside a corporate network when the resident HTTP proxy had a bad day and could not connect to github.com.

Re: January 28th Incident Report

#174

Post by 3minus1 »

jessaustin wrote: post #87
knodi123 wrote: post #55

Earlier quoted context omitted.

> Usually its just cheaper to be down for an hour or two, versus architect for the end of times The opposite of this philosophy was the motivation behind creation of the internet in the first place.

This seems precisely wrong. Some reading: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen... https://www.jwz.org/doc/worse-is-better.html [thanks for the hint 'thinkpad20! I don't know what I was thinking.]

I think he was referring to ARPANET being a military project whose goal was a system that could survive a nuclear attack or other such calamity.

Re: January 28th Incident Report

#175

Post by antirez »

nemothekid wrote: post #7
bosdev wrote: post #2

There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.

Given the dependency in question is Redis, such a solution is probably exacerbated by the fact Redis hasn't really had a decent HA solution. This is also hidden by the fact that Redis is really reliable (in my experience at least). In my experience it usually takes an ops event (like adding more RAM to the redis machine) to realize where a crutch has been developed on Redis in critical paths.

Hello, Sentinel 2 is doing quite a good work for many users, here is for example Flickr report: http://code.flickr.net/2014/07/31/redis-sentinel-at-flickr/

Of course Sentinel does not make Redis conceptually different from what it is from the point of view of consistency guarantees during failures. It performs best-effort attempt to select the best slave to retain writes, but under certain failure modes its possible to lose writes during a failover.

This is common with many failover solutions of *SQL systems as well btw. It depends on your use case if this is an affordable risk or not. For most Redis use cases, usually the risk of losing some writes after certain failovers is not a big issue. For other use cases it is, and a store that retains the writes during all the failure scenarios should be used.

Re: January 28th Incident Report

#176

Post by gaius »

You can very clearly see two kinds of people posting on this thread: those who have actually dealt with failures of complex distributed systems, and those who think it's easy.

Re: January 28th Incident Report

#178

Post by steveklabnik »

Animats wrote: post #144
nissehulth wrote: post #121

Earlier quoted context omitted.

From a developers point of view, you're right. But there are package management systems and other stuff depending on being able to download from Github. Ofc, Github isn't to blame for this, rather the ones that thought Github would be great to use as a CDN.

"But there are package management systems and other stuff depending on being able to download from Github." Rust program building, for example, seems to require that Github be up.

Only if you need to fetch new dependencies.

Re: January 28th Incident Report

#179

Post by xzlzx »

bcook wrote: post #157
ryanlol wrote: post #152

Earlier quoted context omitted.

Why would you feel bad about not being familiar with an abbreviation?

The statement about having poor deductive logic skills was the more insulting part of the post (compared to ignorance of an initialism, which I think you are correct in thinking is insignificant).

Oddly enough, I wasn't stating that the person had poor deductive logic. I was stating the exact steps I took to find the answer myself.

Re: January 28th Incident Report

#180

Post by existencebox »

vacri wrote: post #169
eric_h wrote: post #83

> One of the biggest customer-facing effects of this delay was that status.github.com wasn't set to status red until 00:32am UTC, eight minutes after the site became inaccessible. We consider this to be an unacceptably long delay, and will ensure faster communication to our users in the future. Amazon could learn a thing or two from Github in terms of understanding customer expectations.

> Amazon could learn a thing or two from Github in terms of understanding customer expectations. Do you mean that "the cloud provider that is bigger than the next 14 combined and whose jargon has spread through the community" doesn't understand what customers are interested in and delivering on that?

Gonna speak up to defend OP here: I've worn the devops hat for products across multiple "Large Companies" (Amazon and larger scale) and found that for small products where it was me and a few other devs keeping the lights on, we would have outage alerts on status pages/twitter typically _before_ public users even realized something was wrong, since we were all very high touch on the project.

The bigger a project gets, the less prioritized something like a status page often seems to get. Larger entities certainly _have_ them but I often see more things interfering as scale grows (this isn't only a MS thing, let me make clear) whether it be domain switches between engineering and social management (status is often via twitter), feeding the status page via a long telemetry/monitoring platform that has some lag, or a high threshold for what "outage" means to avoid flappy notices (at the cost of some false negatives).

I'm not even going to make a value judgement on the tradeoff of these costs at this point, (I certainly wouldn't dismiss it offhand as a net negative although equally it's not all roses) but at the very least I'd observe that something like a status page _CAN_ be serviced very well from an up and comer (for as much as Github is that any more) and it's far from a true statement that bigCOs can't take learnings from improving customer happiness from newer entities. (In fact, I wish that was a more common practice!)

Post reply on HN