This happens the day after the whistleblower's 60 Minutes interview? Maybe another rogue (whistleblower is surprised Facebook is a for profit business focused on profit .. not saying good or bad given how big facebook is but really how do you police the world and what side do you take in all the madness/negative parts that make up humanity thus facebook) Facebook employee took it offline ... any chance of that?
Understanding how Facebook disappeared from the internet
181–190 of 280 posts
Re: Understanding how Facebook disappeared from the internet
#182Earlier quoted context omitted.
Unpopular opinion but I think a talent exodus and turnover are likely at fault here as well. The people who stood up this juggernaut are no longer here, and Facebooks ability to consistently attract talent that is competent enough to maintain such a formidable beast is hampered by repetitive revelations that it operates at the net-loss of humanity as a whole. Facebooks no longer an innovator, just a mining operation…
I'd say the same thing about Google.
Yes well lets have this outage again in that flavour in say... 11 months?
Re: Understanding how Facebook disappeared from the internet
#183I'm on 1.1.1.1 and well..... FB was down.
Re: Understanding how Facebook disappeared from the internet
#184Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…
> They are essentially doing glorified bikeshedding, and providing an explainer of how BGP works that can be gleaned by reading virtually any introductory text on it, and using it as a marketing opportunity. I mean, I'm not really sure what the purpose of a corporate blog is except that? You make posts about whatever will garner attention in order to get some views, maybe spread some information, and - of course - tu…
The CEO recently pointed out on Twitter that the primary purpose of the company blog was hiring, and that the company would write with that in mind. The post taught me something I didn’t already know, and left me more impressed than I was before with Cloudflare. Mission accomplished.
Re: Understanding how Facebook disappeared from the internet
#185Earlier quoted context omitted.
Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.
What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out. I suppose with something like BGP it would be very difficult to get such a fall…
Use `iptables-apply` instead.
iptables-apply reverts to previous config instead of flushing all the rules.
Re: Understanding how Facebook disappeared from the internet
#186Earlier quoted context omitted.
I'd say the same thing about Google.
Yes well lets have this outage again in that flavour in say... 11 months?
Taking a bet that complex systems will fail is usually free money :).
Having been a part of Google the only thing more awe inspiring than the sheer complexity of production is the fact that it all worked so well.
This is not a dig at current Googlers but entropy is cruel and uncaring. Perhaps parts of the stack which have been kept fit & fresh in people's minds due to constant rewrites will last longer but there are tons of places in the depot that are unowned despite serving production query traffic and the number of engineers that have any context to support it grows smaller over time.
Re: Understanding how Facebook disappeared from the internet
#187They used the "foot blunderbus".
Re: Understanding how Facebook disappeared from the internet
#188Question about the WARP map - I assume the grey countries are places where Cloudflare doesn't have any presence, but what about Egypt/Oman? Why are they green? And why is Australia orange and not red?
Because most of the outage happened in the early hours of the morning AU time, so not that many people online.
Re: Understanding how Facebook disappeared from the internet
#189Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing gl…
The whole blog post seemed like a click and bait to me.
Nothing is "explained" other than what we knew already, that some unlucky SOB shot themselves in the head with a BGP shotgun.
I understand, they have no way of knowing what happened inside Facebook. But they could give some detail about exactly what the BGP updates were, the structure of the IP space served etc.
Re: Understanding how Facebook disappeared from the internet
#190Question about the WARP map - I assume the grey countries are places where Cloudflare doesn't have any presence, but what about Egypt/Oman? Why are they green? And why is Australia orange and not red?
Also curious about the green part.