Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

241–250 of 306 posts

Re: More details about the October 4 outage

#241

Post by yaknh »

ethbr0 wrote: post #205
bob1029 wrote: post #164

Earlier quoted context omitted.

Relentless customer service is never going to screw you over in my experience... It pains me that we have to constantly play these games of abstraction between engineer and customer. You are presumably working a job which involves some business and some customer. It is not a fucking daycare. If any of my customers are pissed about their experience, I want to be on the phone with them as soon as humanly possible and I…

I think of this is terms of empathy: every engineer should be able to provide a quick and accurate answer to "What do our customers want? And how do they use our product?" I'm not talking esoterica, but at least a first approximation.

Why? Like we all are customers as well as an employee.

Re: More details about the October 4 outage

#242

Post by yaknh »

agwa wrote: post #231
UncleMeat wrote: post #228

Earlier quoted context omitted.

I work at Google and have written more than a few blameless postmortems. You don't need to quote things to me. Is what was described above "finger pointing or shaming"? I don't work in TI so I didn't experience this meeting but it doesn't seem like it is. It also doesn't sound to me like this was the only outcome, where the execs just wagged their fingers at engineers and called it a day. Of course there'd be all sor…

Yes, if I were made to attend a mandatory training in which my leaders read customer emails saying that the outage caused them to lose trust in the company, I would feel ashamed. That was surely the goal of that exercise. The fact that there were also process improvements doesn't make it any less wrong. Thankfully, other comments in this thread suggest that this is not how Google normally does things.

That's fuck up.

Re: More details about the October 4 outage

#243

Post by sverhagen »

dmoy wrote: post #222
Johnny555 wrote: post #43

Earlier quoted context omitted.

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine

The blackout of the northeast US and parts of Canada, in 2003 was really caused by something relatively small. Imagine Facebook, yesterday, causing some weird cascading effect on the power grid, and pulling half of the country with it...

Re: More details about the October 4 outage

#244

Post by bigiain »

plorg wrote: post #17
dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker. A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal t…

> That quantity is about as much as an aluminum smelter switching on or off.

Interestingly, the mountains east of Portland OR, where all the Aluminum smelters used to be, are now full of FAANG datacenters relying on the power infrastructure (and pricing) the Aluminum industry used to use...

https://www.oregonlive.com/silicon-forest/2015/10/small-town...

And Washington state too:

https://www.bizjournals.com/seattle/blog/techflash/2015/11/p...

Re: More details about the October 4 outage

#245

Post by wombatpm »

cecilpl2 wrote: post #195
mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

Human error is a cop-out excuse anyway, since it's not something you can fix going forward. Humans err, and if a mistake was made once it could easily be made again.

To err is human. To really fsk things up you need a computer

Re: More details about the October 4 outage

#246

Post by jauer »

divbzero wrote: post #130
chomp wrote: post #15

So someone ran "clear mpls lsp" instead of "show mpls lsp"?

For context, parent comment is trying to decipher this heavily-PR-reviewed paragraph: > During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these t…

s/assess/augment/g

Re: More details about the October 4 outage

#248

Post by alexchantavy »

toast0 wrote: post #125
pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world…

Thanks for the very helpful explanation!

Re: More details about the October 4 outage

#249

Post by toast0 »

pm2222 wrote: post #215
toast0 wrote: post #198

Earlier quoted context omitted.

> Right I was hoping the DNSs of FB ought to be smarter than usual and let's say when DNS at Seattle-1 cannot reach backbone it'd respond with IP of perhaps NYC/SF before it starts the BGP withdrawal. The problem there is coordination. The PoPs don't generally communicate amongst themselves (and may not have been able to after the FB backbone was broken, although technically, they could have through transit connectiv…

You are to the point and precise. This is exactly the problem. Each individual PoP did the right thing by dropping out of the anycast, but because they all did it, it was the wrong thing. Somehow I feel the design is flawed because if abuses DNS server status a bit. I mean DNS server down and BGP withdrawal for the DNS server is a perfect combination, however connectivity between DNS and backend server down, DNS up a…

I think this is not necessarily a flaw of the design. It's a fundamental weakness of the real world.

Either you can take the backbone being unavailable as a symbol that the PoP is broken, and kill the PoP; or you can take the backbone being unavailable as a symbol that the backbone is broken and do your best.

When either interpretation is wrong, you'll need humans to come around and intervene. It's much more common that the only the PoP is broken, so having that require intervention results in more effort.

The flaw here is more that the intervention required to get the backbone back was hard to do because internal tools to bring back the backbone relied on DNS which relied on the backbone being up. As well, there were some reports that physical security relied on the backbone being up. And that restoring the backbone needed physical access.

This isn't the first largescale FB outage where the root cause was a bad configuration was pushed globally quickly. It's really something they need to learn not to do. But, even without that, being able to get key things running again, like the backbone, and DNS, and the configuration system(s), and centralized authentication, needs to be doable without those key systems running. I suspect at least some of that will be improved on, and hopefully regularly practiced.

Re: More details about the October 4 outage

#250

Post by ativzzz »

yaknh wrote: post #241
ethbr0 wrote: post #205

Earlier quoted context omitted.

I think of this is terms of empathy: every engineer should be able to provide a quick and accurate answer to "What do our customers want? And how do they use our product?" I'm not talking esoterica, but at least a first approximation.

Why? Like we all are customers as well as an employee.

Because we as engineers create software for our customers, and if you don't understand who your customers are how can you create software that actually suits their needs?

Very rarely are we our own customers

Post reply on HN