Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

231–240 of 306 posts

Re: More details about the October 4 outage

#231

Post by agwa »

UncleMeat wrote: post #228
agwa wrote: post #200

Earlier quoted context omitted.

From the SRE book: "For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or te…

I work at Google and have written more than a few blameless postmortems. You don't need to quote things to me. Is what was described above "finger pointing or shaming"? I don't work in TI so I didn't experience this meeting but it doesn't seem like it is. It also doesn't sound to me like this was the only outcome, where the execs just wagged their fingers at engineers and called it a day. Of course there'd be all sor…

Yes, if I were made to attend a mandatory training in which my leaders read customer emails saying that the outage caused them to lose trust in the company, I would feel ashamed. That was surely the goal of that exercise. The fact that there were also process improvements doesn't make it any less wrong.

Thankfully, other comments in this thread suggest that this is not how Google normally does things.

Re: More details about the October 4 outage

#232

Post by Animats »

You can see the security/reliability tradeoff problem here.

You need a control plane. But what does it run over? Your regular data links? A problem if it also controls their configuration. Something outside your own infrastructure, like a modest connection to the local ISP as a backup? That's an attack vector.

One popular solution is to keep both the current and previous generation of the control plane up. Both Google and AT&T seem to have done that. AT&T kept Signalling System 5 up for years after SS7 was doing all the work. Having two totally different technologies with somewhat different paths is helpful.

Re: More details about the October 4 outage

#234

Post by ThinkBeat »

I am curious about something.

It has been quite a while since i had any job that required me to think about DCs.

Back in the day we would have a setup of regular modems.

If all hell broke loose, then we could dial up a modem and have access to the system. It was not fast, and it was a pain, but we could get access that way.

(I am skipping a lot of steps. There was heavy security involved in getting access)

I guess landlines might not be an option anymore??

Re: More details about the October 4 outage

#235

Post by eyelidlessness »

dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

Think Thundering Herd problem on the scale you already know from context. Partial service is a kind of backpressure.

Re: More details about the October 4 outage

#236

Post by mark-r »

_moof wrote: post #176
oarabbus_ wrote: post #107

Earlier quoted context omitted.

And if the secondary access fails, then what? Backup systems are not reliable 100% of the time.

Then you were 1-FT, which is still worlds better than 0-FT. "Don't put two engines on the plane because both of them might fail" is not how fault tolerance works.

I worked with a guy who was an amateur pilot, and he had an opinion about dual engine planes. He said the purpose of the second engine was to get you to the scene of the crash.

Re: More details about the October 4 outage

#237

Post by jrockway »

colanderman wrote: post #203
ghostoftiber wrote: post #115

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

WhatsApp failed similarly, but I thought it was a poor design decision to do so. Anyone waiting on communication through WhatsApp had no indication (outside the media) that it was unavailable, and that they should find a nother communication channel. Don't paper connectivity failures. It disempowers users

I think these features are designed for the timescales like "between subway stations which have cell service" or even "driving into a tunnel". In those cases, it seems totally appropriate to me.

Re: More details about the October 4 outage

#238

Post by timpattinson »

nomel wrote: post #76
dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

If your system is pulling 500 watts at 120V, that's around 4A of line voltage. If you drop down 20% to 100V, the output will happily still pull its regulated voltage, but now the line components are seeing ~20% more, at 5A. For brown out, you need to overrate your components, and/or shut everything off if the line voltage goes too low. I used to do electrical compliance testing in a previous life, with brown out test…

"output will happily still pull its regulated voltage" you mean power, right?

Re: More details about the October 4 outage

#239

Post by nbm »

1vuio0pswjnm7 wrote: post #230

"Those translation queries are answered by our authoritative name servers that occupy well known IP addresses themselves, which in turn are advertised to the rest of the internet via another protocol called the border gateway protocol (BGP)." "To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy…

Perhaps the right word is “system” - the combination of software (DNS protocol server software, health checker, BGP agent, and so forth) involved in making the DNS service available (or not, in this case). These could be running on the same computer, or separate ones if you are particularly imaginative.

Unfortunately, “DNS system” means something very different than if you said “load balancing system”, so “server” is simpler.

(Usual disclaimer: work at FB, even on this exact stuff, but not representing it.)

Re: More details about the October 4 outage

#240

Post by eddieroger »

Animats wrote: post #232

You can see the security/reliability tradeoff problem here. You need a control plane. But what does it run over? Your regular data links? A problem if it also controls their configuration. Something outside your own infrastructure, like a modest connection to the local ISP as a backup? That's an attack vector. One popular solution is to keep both the current and previous generation of the control plane up. Both Googl…

The post mentioned that the out-of-band network was also down, and I’m curious what that entails and how it was also impacted. They must not have been on external DNS or had static IPs to access recovery. I’m sure they won’t share more than this now, but I’d sure love to hear more about the OOB access.
Post reply on HN