More details about the October 4 outage • Hacker News Board

Re: More details about the October 4 outage

#271

Post by blackbear_ » Wed, Oct 06, 2021, 9:32 AM UTC

hef19898 wrote: post #262

tialaramex wrote: post #256

Earlier quoted context omitted.

Organisations have a bad habit of using "human error" to blame systemic problems whose true root cause is inadequate leadership on individual low level employees. So, we're glad to see Facebook didn't try this shitty practice. For a modern example, look for information on Symantec's "A tough day as leaders" in which they try to blame an incident that's clearly a result of at least incompetence by senior management on…

Boeing's initial response to the 737MAX crashes comes to mind as well.

And the Volkswagen emissions scandal too

Re: More details about the October 4 outage

#272

Post by Bad_CRC » Wed, Oct 06, 2021, 9:32 AM UTC

BurningFrog wrote: post #229

tobr wrote: post #137

Earlier quoted context omitted.

The buggy audit tool was probably made by a human too, though.

It's humans all the way down!

something something soylent green

Re: More details about the October 4 outage

#273

Post by Bad_CRC » Wed, Oct 06, 2021, 9:32 AM UTC

tobr wrote: post #137

mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

The buggy audit tool was probably made by a human too, though.

"hey let's try this github copilot thingy to write an audit tool"

Re: More details about the October 4 outage

#274

Post by fragmede » Wed, Oct 06, 2021, 9:35 AM UTC

ativzzz wrote: post #250

yaknh wrote: post #241

Earlier quoted context omitted.

Why? Like we all are customers as well as an employee.

Because we as engineers create software for our customers, and if you don't understand who your customers are how can you create software that actually suits their needs? Very rarely are we our own customers

I would argue that SREs are consistently our own customers in a way unique to SRE.

Re: More details about the October 4 outage

#275

Post by hef19898 » Wed, Oct 06, 2021, 9:53 AM UTC

blackbear_ wrote: post #271

hef19898 wrote: post #262

Earlier quoted context omitted.

Boeing's initial response to the 737MAX crashes comes to mind as well.

And the Volkswagen emissions scandal too

Yes! I forgot they initially tried to blame it at some low-level engineer. There is a old joke in Germany, predating the VW scandal, saying that is definitely was the night guard.

Re: More details about the October 4 outage

#276

Post by dx034 » Wed, Oct 06, 2021, 9:53 AM UTC

slim wrote: post #162

chomp wrote: post #15

So someone ran "clear mpls lsp" instead of "show mpls lsp"?

My guess : he executed the command from shell history. Commands look similar enough and he hit Enter too quickly

But why would the clear command be in their history?

Re: More details about the October 4 outage

#277

Post by MertsA » Wed, Oct 06, 2021, 9:59 AM UTC

submain wrote: post #174

thirtyseven wrote: post #16

Earlier quoted context omitted.

With an outage this big, even a post for a technical audience will get read by non-technical people (including journalists), so I'm sure it helps to include details like this.

The media: "Facebook engineer typed command. This is what happened next."

"10 backbone router commands Mark Zuckerberg doesn't want you to know. Number 7 will shock you!"

Re: More details about the October 4 outage

#278

Post by iso1210 » Wed, Oct 06, 2021, 10:06 AM UTC

wombatpm wrote: post #245

cecilpl2 wrote: post #195

Earlier quoted context omitted.

Human error is a cop-out excuse anyway, since it's not something you can fix going forward. Humans err, and if a mistake was made once it could easily be made again.

To err is human. To really fsk things up you need a computer

To err requires a computer. To really fsk things up requires automation.

Re: More details about the October 4 outage

#279

Post by izietto » Wed, Oct 06, 2021, 10:06 AM UTC

tobr wrote: post #137

mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

The buggy audit tool was probably made by a human too, though.

No they used GitHub Copilot

Re: More details about the October 4 outage

#280

Post by nuerow » Wed, Oct 06, 2021, 10:12 AM UTC

hef19898 wrote: post #262

tialaramex wrote: post #256

Earlier quoted context omitted.

Organisations have a bad habit of using "human error" to blame systemic problems whose true root cause is inadequate leadership on individual low level employees. So, we're glad to see Facebook didn't try this shitty practice. For a modern example, look for information on Symantec's "A tough day as leaders" in which they try to blame an incident that's clearly a result of at least incompetence by senior management on…

Boeing's initial response to the 737MAX crashes comes to mind as well.

> Boeing's initial response to the 737MAX crashes comes to mind as well.

Truth be told, Boeing's response to the 737MAX crashes was to blame people working for other organizations, thus the blame would not fall within neither Boeing engineers/technicians nor the Boeing organization. That's a total and complete cop out.

Pinning the blame on a company employee at least implies that the company itself has some responsibility.