Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

91–100 of 306 posts

Re: More details about the October 4 outage

#91

Post by detaro »

packetslave wrote: post #72
vitus wrote: post #48

Earlier quoted context omitted.

With something as fundamental as the network, no way around it. - Okay, we'll set up a separate maintenance network in case we can't get to the regular network. - Wait, but we need a maintenance network for the maintenance network...

"Okay, we'll pull in a DSL line from a completely separate ISP for the out-of-band access." (guess what else is in that manhole/conduit?) "Okay, we'll use LTE for out-of-band!" (oops, the backhaul for the cell tower goes under the same bridge as the real network) True diversity is HARD (not unsolvable, just hard. especially at scale)!

Although the difference here is that loosing connection and out-of-band for a single data center shouldn't be as catastrophic for Facebook, so your examples would be tolerable?

Re: More details about the October 4 outage

#92

Post by riffic »

Hokusai wrote: post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

I'm reading your comment as a form of "feigning surprise", in other words a statement along the lines of "I can't believe target audience doesn't know about x concept".

more on the concept: https://noidea.dog/blog/admitting-ignorance

Re: More details about the October 4 outage

#93

Post by dreamlayers »

nomel wrote: post #76
dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

If your system is pulling 500 watts at 120V, that's around 4A of line voltage. If you drop down 20% to 100V, the output will happily still pull its regulated voltage, but now the line components are seeing ~20% more, at 5A. For brown out, you need to overrate your components, and/or shut everything off if the line voltage goes too low. I used to do electrical compliance testing in a previous life, with brown out test…

Why wouldn't the power supply shut down due to overcurrent protection?

Re: More details about the October 4 outage

#95

Post by 14 »

tpmx wrote: post #84
jd3 wrote: post #39

Earlier quoted context omitted.

Was this ever confirmed? NYT tech reporter Mike Isaac issued a correction to his previous reporting about it. > the team dispatched to the Facebook site had issues getting in because of physical security but did not need to use a saw/ grinder. https://twitter.com/MikeIsaac/status/1445196576956162050

> so.....they had to use a jackhammer, got it

This. We know they lie so have to assume the worst when they say something. The correction only says saw/angle grinder. “No tools were needed” is a little clearer. However I am not sure why it matters if they used a key or a grinder?

Re: More details about the October 4 outage

#96

Post by 88913527 »

Hokusai wrote: post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

There are plenty of technical people (or people employed in technical roles) who don't understand how DNS works. For example, I field questions on why "hostname X only works when I'm on VPN" at work.

Re: More details about the October 4 outage

#97

Post by detaro »

dreamlayers wrote: post #93
nomel wrote: post #76

Earlier quoted context omitted.

If your system is pulling 500 watts at 120V, that's around 4A of line voltage. If you drop down 20% to 100V, the output will happily still pull its regulated voltage, but now the line components are seeing ~20% more, at 5A. For brown out, you need to overrate your components, and/or shut everything off if the line voltage goes too low. I used to do electrical compliance testing in a previous life, with brown out test…

Why wouldn't the power supply shut down due to overcurrent protection?

That's likely on the output side, which doesn't see overcurrent.

Re: More details about the October 4 outage

#98

Post by b112 »

vitus wrote: post #70
jeffbee wrote: post #55

Earlier quoted context omitted.

Google also had a runaway automation outage where a process went around the world "selling" all the frontend machines back to the global resource pool. Nobody was alerted until something like 95% of global frontends had disappeared. This was an important lesson for SREs inside and outside Google because it shows the dangers of the antipattern of command line flags that narrow the scope of an operation instead of expa…

Agreed, but with an amendment: If your tool is capable of draining the whole universe, period , it is too dangerous to exist. That was one of the big takeaways: global config changes must happen slowly. (Whether we've fully internalized that lesson is a different matter.)

If your tool is capable of draining the whole universe

Why did I think of humans, when I read this. :P

Re: More details about the October 4 outage

#99

Post by gist »

Hokusai wrote: post #9

> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel l…

> What is the target audience of this post?

Separate point to your question.

FB is under no obligation to provide more details than they need to because a small segment of the population (certainly relative to their 'customers') might find it interesting or helpful or entertaining. FB is a business. They can essentially do (and should be able to) do whatever they want. There is no requirement (and there should be no requirement) to provide the general public with more info than they want subject to any legal requirement. If the government wants more (and are entitled to more info) they can ask for it and FB can decide if they are required to comply.

FB is a business. Their customers are not the tech community looking to get educated and avoid issues themselves at their companies or (as mentioned) be entertained. And their customers (advertisers or users) can decide if they want to continue to patronize FB.

I always love on HN seeing 'hey where is the post mortem' as if it's some type of defacto requirement to air dirty laundry to others.

If I go to the store and there is not paper towels there I don't need to know why there are no towels and what the company will do going forward to prevent any errors that caused the lack of that product. I can decide to buy another brand or simply take steps to not have it be an issue.

Re: More details about the October 4 outage

#100

Post by elzbardico »

temp_praneshp wrote: post #74
Ansil849 wrote: post #67

Earlier quoted context omitted.

Going on at lengths about how the trade off between prolonged downtime and strict security protocols is a worthy trade off is erecting a nonsensical strawman, the literal definition of spinning a story. The key issue had nothing to do with Facebook's data center security protocols.

No one except you is erecting a nonsensical strawman.

The FB PR army is at full force today
Post reply on HN