Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

261–270 of 306 posts

Re: More details about the October 4 outage

#261

Post by scrollaway »

perryizgr8 wrote: post #253
ghostoftiber wrote: post #115

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

On Instagram I was even able to "like" posts while the outage was in effect. Not sure if the app replayed those when the service came back.

I suspect so. From the app's point of view it's likely not terribly different to losing cell service for a while.

Re: More details about the October 4 outage

#262

Post by hef19898 »

tialaramex wrote: post #256
nuerow wrote: post #142

Earlier quoted context omitted.

> I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error. Wouldn't human error reflect extremely poorly on the company though? I mean, for human error to be the root cause of this mega-outage, that would imply that the company's infrastructure and operational and security practices were so ineffective that a single person scre…

Organisations have a bad habit of using "human error" to blame systemic problems whose true root cause is inadequate leadership on individual low level employees. So, we're glad to see Facebook didn't try this shitty practice. For a modern example, look for information on Symantec's "A tough day as leaders" in which they try to blame an incident that's clearly a result of at least incompetence by senior management on…

Boeing's initial response to the 737MAX crashes comes to mind as well.

Re: More details about the October 4 outage

#263

Post by mst »

toast0 wrote: post #249
pm2222 wrote: post #215

Earlier quoted context omitted.

You are to the point and precise. This is exactly the problem. Each individual PoP did the right thing by dropping out of the anycast, but because they all did it, it was the wrong thing. Somehow I feel the design is flawed because if abuses DNS server status a bit. I mean DNS server down and BGP withdrawal for the DNS server is a perfect combination, however connectivity between DNS and backend server down, DNS up a…

I think this is not necessarily a flaw of the design. It's a fundamental weakness of the real world. Either you can take the backbone being unavailable as a symbol that the PoP is broken, and kill the PoP; or you can take the backbone being unavailable as a symbol that the backbone is broken and do your best. When either interpretation is wrong, you'll need humans to come around and intervene. It's much more common t…

I'm not going to claim to be a BGP expert, but as I understand it the way BGP propagation tends to work makes it a pretty global thing just in terms of how the router hardware handles stuff, which makes it unusually tricky to avoid.

I don't disagree about the general problem mind, I just have a feeling that fixing "don't push configs globally" for BGP specifically is unusually complicated.

Re: More details about the October 4 outage

#264

Post by fragmede »

bryan_w wrote: post #108
chomp wrote: post #15

So someone ran "clear mpls lsp" instead of "show mpls lsp"?

Seems like it. It's kinda like typing hostname and accidentally poking your yubikey (Not that I've done that...) or the date command that both let's you set the date and format the date

  ykpersonalize -u -1 -o -fast-trig
(or some variation thereof)

Re: More details about the October 4 outage

#265

Post by avh02 »

colanderman wrote: post #203
ghostoftiber wrote: post #115

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

WhatsApp failed similarly, but I thought it was a poor design decision to do so. Anyone waiting on communication through WhatsApp had no indication (outside the media) that it was unavailable, and that they should find a nother communication channel. Don't paper connectivity failures. It disempowers users

whatsapp used to have a "status" page in their app, they removed it years ago though.

That said - yeah, i figured there's an issue when hard refreshing whatsapp web (sometimes has a kink that needs to be refreshed away) didn't work.

Re: More details about the October 4 outage

#266

Post by [deleted] »

vitus wrote: post #70
jeffbee wrote: post #55

Earlier quoted context omitted.

Google also had a runaway automation outage where a process went around the world "selling" all the frontend machines back to the global resource pool. Nobody was alerted until something like 95% of global frontends had disappeared. This was an important lesson for SREs inside and outside Google because it shows the dangers of the antipattern of command line flags that narrow the scope of an operation instead of expa…

Agreed, but with an amendment: If your tool is capable of draining the whole universe, period , it is too dangerous to exist. That was one of the big takeaways: global config changes must happen slowly. (Whether we've fully internalized that lesson is a different matter.)

[deleted]

Re: More details about the October 4 outage

#268

Post by Cthulhu_ »

mumblemumble wrote: post #132

> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.

I wouldn't be surprised if that tool was a shell script with a mistyped conditional somewhere, I really dislike shell scripting.

Re: More details about the October 4 outage

#269

Post by Cthulhu_ »

bigiain wrote: post #244
plorg wrote: post #17

Earlier quoted context omitted.

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker. A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal t…

> That quantity is about as much as an aluminum smelter switching on or off. Interestingly, the mountains east of Portland OR, where all the Aluminum smelters used to be, are now full of FAANG datacenters relying on the power infrastructure (and pricing) the Aluminum industry used to use... https://www.oregonlive.com/silicon-forest/2015/10/small-town... And Washington state too: https://www.bizjournals.com/seattle/bl…

That's pretty interesting, I'm sure those aluminium foundries would need to be careful about turning the power on as well.

Tangentially related, aluminium production in the Netherlands may shut down soon; because of a sudden spike in gas prices (due to mismanagement), electricity prices have also gone up, making producing aluminium no longer cost-effective. €2400 in electricity to produce a ton of aluminium worth €2500 kinda cost effectiveness.

I wouldn't be surprised if the big datacenters here will try and offload some of their workloads to datacenters elsewhere with lower energy costs. Mind you, I'm pretty sure these datacenters make long-running deals on electricity prices.

Re: More details about the October 4 outage

#270

Post by fragmede »

vitus wrote: post #70
jeffbee wrote: post #55

Earlier quoted context omitted.

Google also had a runaway automation outage where a process went around the world "selling" all the frontend machines back to the global resource pool. Nobody was alerted until something like 95% of global frontends had disappeared. This was an important lesson for SREs inside and outside Google because it shows the dangers of the antipattern of command line flags that narrow the scope of an operation instead of expa…

Agreed, but with an amendment: If your tool is capable of draining the whole universe, period , it is too dangerous to exist. That was one of the big takeaways: global config changes must happen slowly. (Whether we've fully internalized that lesson is a different matter.)

SREs live in a dangerous world, unfortunately. It's entirely possible the "tool" in question is a shell script that gets fed a list of bad cells but some bug causes it to get a list of all the cells instead.

Some tools are well engineered, capable of the Sisyphean task of globally deploying updates but others are rapid prototypes that, sure, are too dangerous to exist, but the whole point of SREs being capable programmers is that the work has problems that are most efficiently solved with one-off code that just isn't (because it can't be) rigorously tested before being used. You can bet there was some of that used in recovering from this incident. (I'm sure there were many eyes reviewing the code before being run, but that only goes so far when you're trying to do something that you never expected, like having to revive Facebook.)

Post reply on HN