Can someone explain why the optimizer would split one route into two? Wouldn’t it be more optimized to coalesce routes whenever possible?
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
71–80 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#72Earlier quoted context omitted.
We thought we did. And tried both public and private lines of communication — without reply. Still waiting.
Hats off to Verizon for treating everyone equally without discrimination.
"We don't care. We don't have to. We're the phone company." - Verizon, probably
It's good to know that Net Neutrality isn't dead.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#73Earlier quoted context omitted.
It doesn't need government intervention. It needs other companies to hold them accountable.
Absolutely. If you are a Verizon service provider customer, please call them and register your feelings on this matter. Make sure you let them know in no uncertain terms that you are considering switching providers based on their lack of following best practices.
And will do so, once another provider becomes available in your area.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#74Earlier quoted context omitted.
We thought we did. And tried both public and private lines of communication — without reply. Still waiting.
Hats off to Verizon for treating everyone equally without discrimination.
Haha hats off to you for a string of comments on this thread that have me lol’ing
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#75One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.
We thought we did. And tried both public and private lines of communication — without reply. Still waiting.
The lack of communication is the bit that justified blog scorching for me.
If everyone was working in good faith, professional courtesy.
But not being able to get someone on the phone? Wtf?
Glad the original provider was responsive and able to resolve. And hats off to your poor reliability engineers today.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#76Every time I see the BGP abbreviation it's about a huge fuck-up. Either somebody hijacks routes intentionally or something like this happens.
Not trying to defend a legacy protocol, but it's one of these cases where basic infrastructure is not newsworthy when it works just fine, like with plumbing.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#77Here's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.
Amen. We need a support group.
"Hi, I'm Teejmya, and I was on call last night"
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#78Earlier quoted context omitted.
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…
The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.
Blame is due where blame is due.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#79Earlier quoted context omitted.
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…
The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.
I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#80Earlier quoted context omitted.
Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…
Very nice writeup on RPKI! I don't know anything about network engineering, but it appears that RPKI will distribute trust from ISPs to RIRs (Regional Internet Registries) like ARIN and RIPE. As I understand it, the RIR will sign your IP allocation with RPKI, which means fat-fingering on your side will result in the ISP not finding you as it takes BGP announcement and RIR confirmation for the ISP to acknowledge your IP. Again, very nice and understandable writeup :)
I guess this does shift the burden of trust from an ISP to the RIR, and the blog post mentions international law as RIR and ISP memberships can be part of different countries and only RIRs would know who has what IP address since only they are TAs (which empowers certain governments over others). So I guess the debate is whether the pain of BGP route leaks and such is greater than the stress of another country having your RPKI entry.
I guess we'll just have to see how badly Verizon messes up in the future.