More details about the October 4 outage • Hacker News Board

Re: More details about the October 4 outage

#201

Post by Johnny555 » Tue, Oct 05, 2021, 9:04 PM UTC

cesarb wrote: post #179

Johnny555 wrote: post #50

Earlier quoted context omitted.

I don't see how suddenly running more traffic is going to trip datacenter breakers -- I could see how flipping on power to an entire datacenter's worth of servers could cause a spike in electrical demand that the power infrastructure can't handle, but if suddently running CPU's at 100% trips breakers, then it seems like that power infrastructure is undersized? This isn't a case where servers were powered off, they we…

The key word is "suddenly". In the electricity grid, demand and generation must always be precisely matched (otherwise, things burn up). This is done by generators automatically ramping up or down whenever the load changes. But most generators cannot change their output instantly; depending on the type of generator, it can take several minutes or even hours to respond to a large change in the demand. Now consider tha…

I get that lots of servers can add up to lots of power, but what is a "lot"? Is 1MW really enough demand to destabilize a regional power grid?

Re: More details about the October 4 outage

#202

Post by lozenge » Tue, Oct 05, 2021, 9:05 PM UTC

agwa wrote: post #150

vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1]. [1] https://sre.google/sre-book/postmortem-culture/

I assume it was training for all SREs, like "this is why we're all doing so much to prevent it from reoccurring"

Re: More details about the October 4 outage

#203

Post by colanderman » Tue, Oct 05, 2021, 9:06 PM UTC

ghostoftiber wrote: post #115

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

WhatsApp failed similarly, but I thought it was a poor design decision to do so. Anyone waiting on communication through WhatsApp had no indication (outside the media) that it was unavailable, and that they should find a nother communication channel.

Don't paper connectivity failures. It disempowers users

Re: More details about the October 4 outage

#204

Post by ethbr0 » Tue, Oct 05, 2021, 9:10 PM UTC

vitus wrote: post #70

jeffbee wrote: post #55

Earlier quoted context omitted.

Google also had a runaway automation outage where a process went around the world "selling" all the frontend machines back to the global resource pool. Nobody was alerted until something like 95% of global frontends had disappeared. This was an important lesson for SREs inside and outside Google because it shows the dangers of the antipattern of command line flags that narrow the scope of an operation instead of expa…

Agreed, but with an amendment: If your tool is capable of draining the whole universe, period , it is too dangerous to exist. That was one of the big takeaways: global config changes must happen slowly. (Whether we've fully internalized that lesson is a different matter.)

As FB opines at the end, at some point, it's a trade-off between power (being able access / do everything quickly) and safety (having speed bumps that slow larger operations down).

The pure takeaway is probably that it's important to design systems where "large" operations are rarely required, and frequent ops actions are all "small."

Because otherwise, you're asking for an impossible process (quick and protected).

Re: More details about the October 4 outage

#205

Post by ethbr0 » Tue, Oct 05, 2021, 9:14 PM UTC

bob1029 wrote: post #164

UncleMeat wrote: post #158

Earlier quoted context omitted.

"Blameless Postmortem" does not mean "No Consequences", even if people often want to interpret it that way. If an organization determines that a disconnect between ground work and a customer's experience is a contributing factor to poor decision making then they might conclude that making engineers more emotionally invested in their customers could be a viable path forward.

Relentless customer service is never going to screw you over in my experience... It pains me that we have to constantly play these games of abstraction between engineer and customer. You are presumably working a job which involves some business and some customer. It is not a fucking daycare. If any of my customers are pissed about their experience, I want to be on the phone with them as soon as humanly possible and I…

I think of this is terms of empathy: every engineer should be able to provide a quick and accurate answer to "What do our customers want? And how do they use our product?"

I'm not talking esoterica, but at least a first approximation.

Re: More details about the October 4 outage

#206

Post by variant » Tue, Oct 05, 2021, 9:14 PM UTC

silicon2401 wrote: post #139

vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> leadership read out emails from customers telling us how we let them down and lost their trust). That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.

I just read your comment out loud if it helps.

Re: More details about the October 4 outage

#207

Post by ddalex » Tue, Oct 05, 2021, 9:17 PM UTC

agwa wrote: post #150

vitus wrote: post #18

Google had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at F…

> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1]. [1] https://sre.google/sre-book/postmortem-culture/

Not the original googler responding, but I have never experienced what they describe.

Postmortems are always blameless in the sense that "Somebody fat fingered it" is not an acceptable explanation for the causes of an incident - the possibility to fat finger it in the first place must be identified and eliminated.

Opinions are my own, as always

Re: More details about the October 4 outage

#208

Post by vitus » Tue, Oct 05, 2021, 9:28 PM UTC

ddalex wrote: post #207

agwa wrote: post #150

Earlier quoted context omitted.

> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1]. [1] https://sre.google/sre-book/postmortem-culture/

Not the original googler responding, but I have never experienced what they describe. Postmortems are always blameless in the sense that "Somebody fat fingered it" is not an acceptable explanation for the causes of an incident - the possibility to fat finger it in the first place must be identified and eliminated. Opinions are my own, as always

> Not the original googler responding, but I have never experienced what they describe.

I have also never experienced this outside of this single instance. It was bizarre, but tried to reinforce the point that something needed to change -- it was the latest in a string of major customer-facing outages across various parts of TI, potentially pointing to cultural issues with how we build things.

(And that's not wrong, there are plenty of internal memes about the focus on building new systems and rewarding complexity, while not emphasizing maintainability.)

Usually mandatory trainings are things like "how to avoid being sued" or "how to avoid leaking confidential information". Not "you need to follow these rules or else all of Cloud burns down; look, we're already hemorrhaging customer goodwill."

As I said, there was significant scar tissue associated with this event, probably caused in large part by the initial reaction by leadership.

Re: More details about the October 4 outage

#209

Post by chasd00 » Tue, Oct 05, 2021, 9:28 PM UTC

packetslave wrote: post #72

vitus wrote: post #48

Earlier quoted context omitted.

With something as fundamental as the network, no way around it. - Okay, we'll set up a separate maintenance network in case we can't get to the regular network. - Wait, but we need a maintenance network for the maintenance network...

"Okay, we'll pull in a DSL line from a completely separate ISP for the out-of-band access." (guess what else is in that manhole/conduit?) "Okay, we'll use LTE for out-of-band!" (oops, the backhaul for the cell tower goes under the same bridge as the real network) True diversity is HARD (not unsolvable, just hard. especially at scale)!

heh i toured a large data center here in dallas and listened to them brag about all the redundant connectivity they had while standing next to the conduit where they all entered the building. One person, a pair of wire cutters, and 5 seconds and that whole datacenter is dark.

Re: More details about the October 4 outage

#210

Post by cnst » Tue, Oct 05, 2021, 9:30 PM UTC

mbauman wrote: post #199

cnst wrote: post #182

Earlier quoted context omitted.

Exactly. And it would actually be worse, because the clients would have to wait for a timeout, instead of simply returning a name error right away.

How would've it been worse? Waiting for a timeout is a good thing as it prevents a thundering herd of refresh-smashing (both automated and manual). I don't know BGP well, but it seems easier for peers to just drop FB's packets on the floor than deal with a DNS stampede.

An average webpage today is several megabytes in size.

How would a few bytes over a couple of UDP packets for DNS have any meaningful impact on anyone's network? If anything, things fail faster, so, there's less data to transmit.

For example, I often use ordns.he.net as an open recursive resolver. They use PowerDNS as their software. PowerDNS has the default of packetcache-servfail-ttl of 60s. OTOH, fb.com A response currently has a TTL of 300s — 5 minutes. So, basically, FB's DNS is cached for roughly the same time whether or not they're actually online.