201–210 of 291 posts
Earlier quoted context omitted.
Not really. You don't have to be a massive player to screw things up with BGP.
No doubt. But it’s 2019 and I can’t muster up much sympathy for a tier 1 who can’t get inbound filters and a responsive NOC implemented correctly - things which were table stakes in 2009.
From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…
Cloudflare reached out multiple times in multiple ways to Verizon, to attempt to resolve the situation. More than eight hours on, after utilising everything from what they were told was a Tier 1 support line to Twitter, they have nothing. Even if we're kind to Verizon about the network failure, which was a global issue, they haven't done anything or said anything to suggest that Cloudflare should be treating them kin…
>"Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons"
Oh the Cloudflare marketing machine. Largest by "utilized"? What does that even mean? Cloudflare is not a Tier 1, a Tier 2, or a major eyeball network. They are pretty far down in the pecking order despite what your marketing department wants us to believe. There's always some fuzzy stat isn't there?
Being too inundated to respond to everyone on the day of outage is a human resource problem, plain and simple The fact that you have taken this so personally is kind of embarrassing. What this blog post, the opportunistic marketing ploy and finger pointing have shown is a complete lack of maturity on your part. You want to call out Verizon for their behavior yet your own behavior is unnecessarily aggressive.
Earlier quoted context omitted.
Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…
Earlier quoted context omitted.
It was gross negligence that leads to this. Nothing more.
Do you work for Verizon's NOC or Network Engineering department then? You have inside knowledge that it was negligence? Because I provided a specific scenario where it would not be "gross negligence."
No, you didn't. You provided a vague conjecture for how the initial cause of the problem might not have been gross negligence, but offered no hypothesis for why Verizon isn't answering the Red Phone.
Earlier quoted context omitted.
To be frank, your post makes it clear that you don't know the context. CF simply cannot do anything on their own to mitigate the problem where Verizon constructs bad BGP routes to Cloudflare IPs and then advertises those routes to third parties. The only mitigation possible is to contact whoever's advertising the bad routes and get them to stop.
Have you read Cloudflare's multiple blog posts regarding BGP? Did you read the tweets from their directors talking about how other customers were unaffected by the event today because of the mitigations put in place? Did you even do the simplest Google about BGP protocols and the plans in place to prevent this from happening in the future? If you're going to try to impose yourself as the gatekeeper of "knowing the co…
Earlier quoted context omitted.
Is it time to put an HF ham radio rig in each major provider’s office? I shudder thinking of a major outage where even phone communication can’t take place. I’m only half joking. Edit: and maybe we just give Verizon a toy walkie talkie
You joke, but cascade failure ain't no laughing matter, either. Emergency plans aren't for fair weather, they are for sh!tstorms. I don't want to say "we are too reliant on the internet" because it's cliche and connectivity is just part of growing the modern world. But we sure as heck need several layers of backup plans in case things go sideways. I, for one, hope there is a secret society of HAMs lurking as mild-man…
Unfortunately, Verizon is one of those networks that isn't present there. But many other networks are represented there and it provides a direct path to those who have config/enable access on some of the largest networks out there. Cuts out the having to go via formal escalation paths and NOC groups that require a trouble ticket before you can engage them.
Earlier quoted context omitted.
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…
Note that RPKI won’t prevent outages caused by route leaks (because the leak has a valid signed origin).
The fact that the original AS Origin is included here makes this even more weaponized.
Brings it back to why doesn't the Noction platform "dirty" the injected announcements. For example, throwing out some Private ASNs or ASNs of "tier 1" providers to prevent those announcements from ever getting propagated around.
Earlier quoted context omitted.
Yes, apologies and thank you! If you email me (matthewatcloudflaredotcom) your shirt size, preference for men's or women's cut, and your postal address with the subject line: "Verizon BGP Leak On Call Support Group" I'll send you a Cloudflare tshirt. Least we can do.
I think somebody should make some "I Fixed the Internet after Verizon Broke It. 20190624" T-shirts.