More details about the October 4 outage
engineering.fb.com
More details about the October 4 outage
1–10 of 306 posts
Re: More details about the October 4 outage
#2Yesterday's post from Facebook
https://engineering.fb.com/2021/10/04/networking-traffic/out...
Re: More details about the October 4 outage
#3It's a no apologies messages:
"We failed, our processes failed, our recovery process only partially worked, we celebrate failure. Our investors were not happy, our users were not happy, some people probably ended in physically dangerous situations due to WhatsApp being unavailable but it's ok. We believe a tradeoff like this is worth it."
- Your engineering team.
Re: More details about the October 4 outage
#4Interesting bit on recovery w.r.t. the electrical grid
> flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ...
I wish there was a bit more detail in here. What's the worst case there? Brownouts, exploding transformers? Or less catastrophic?
Re: More details about the October 4 outage
#5> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally
Imagine being this person.
Tomorrow on /r/tifu.
Re: More details about the October 4 outage
#6It's a no apologies messages: "We failed, our processes failed, our recovery process only partially worked, we celebrate failure. Our investors were not happy, our users were not happy, some people probably ended in physically dangerous situations due to WhatsApp being unavailable but it's ok. We believe a tradeoff like this is worth it." - Your engineering team.
Literally the first line of their first article.[0] This article is just a technical explanation.
[0] https://engineering.fb.com/2021/10/04/networking-traffic/out...
Re: More details about the October 4 outage
#7> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally Imagine being this person. Tomorrow on /r/tifu.
Hahahahahahahaha
Re: More details about the October 4 outage
#8Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…
Likely tripping breakers or overload protection on UPSes?
Often PDUs used in a rack can be configured to start servers up in a staggered pattern to avoid a surge in demand for these reasons.
I'd imagine there's more complications when you're doing an entire DC vs just a single rack, though.
Re: More details about the October 4 outage
#9> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses.
What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel like I'm missing something.
Re: More details about the October 4 outage
#10> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally Imagine being this person. Tomorrow on /r/tifu.
"It was the intern" is only cute once.