141–150 of 280 posts
Earlier quoted context omitted.
It wouldn't be the first time it's happened, and you would expect their network and security folk to understand single points of failure.
You would think those in charge of security would know better with their mid-six figure salaries. But then again these days they stopped hiring based on merit but rather certain other “metrics” no wonder they were so incompetent...
All very amusing. But most amusing was that facebook employees' badges wouldn't work. Hilarious. Can't get in the office if facebook.com is not reachable. (or whatever the unreachable service was)
No key-card access means even non-need-to-know internal employees wouldn't see the deed, and plausible deniability is spawned for everyone else.
Earlier quoted context omitted.
No surprises here. They do it all the time for websites they cdn. Cloudflare might as well be a marketing company for how much they puff about themselves.
Cloudflare is also incredibly dodgy with their billing practices which you wouldn't pick up from their marketing. If you downgrade from a Paid to a Free account they automatically and silently add a paid add-on for 10 Page Rules.
> If you do not want to be charged for the additional page rules, you should ensure you only have 3 active rules before you downgrade from Pro to Free.
[1] https://community.cloudflare.com/t/i-canceled-my-subscriptio...
Earlier quoted context omitted.
You would think those in charge of security would know better with their mid-six figure salaries. But then again these days they stopped hiring based on merit but rather certain other “metrics” no wonder they were so incompetent...
Can you elaborate on this? Curious about how they hire their 500k people.
Some other internal issues could have broken many systems, including BGP.
They could have had some sort of internal systems failure, and intentionally withdrew BGB to cut off the flood of connection attempts (making recovery easier).
Many possibilities.
All very amusing. But most amusing was that facebook employees' badges wouldn't work. Hilarious. Can't get in the office if facebook.com is not reachable. (or whatever the unreachable service was)
Far fetched, obviously, but.... Jurassic Park, Dennis Nedry-type situation? Shut down the master systems to hide the theft (or in this case, destruction) of vital info. No key-card access means even non-need-to-know internal employees wouldn't see the deed, and plausible deniability is spawned for everyone else.
Earlier quoted context omitted.
What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out. I suppose with something like BGP it would be very difficult to get such a fall…
This is a key feature of Junos on Juniper devices. It's called 'commit confirmed' and it will apply the config and then auto-rollback if you don't confirm it within a certain amount of time. https://www.juniper.net/documentation/us/en/software/junos/c... But Facebook's network is certainly much more complex and automated than just doing one commit on one device. But I think they do still uses Juniper devices at the e…
Software defined networking. In PHP.
Earlier quoted context omitted.
Timezones are the most annoying thing... right after encoding
I personally find timezones more annoying. At least with encoding once you figure things out it will work indefinitely. Timezones can simply change from under you with or without notice.
(And that's not counting daylight savings time, and its varied observance!)
This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?