Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
61–70 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#62> The RPKI framework that we implemented and deployed globally last year is designed to prevent this type of leak. It enables filtering on origin network and prefix size. The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is.
Does RPKI prevent Cloudflare from announcing additional /22 routes during an incident like this? Any network with RPKI implemented would reject the /22s, but those who ignore it should pick them up over the leaked /21s.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#63I am surprised that CF is as aggressive toward Verizon in public as they are. Once you start breaking the Internet for stupid reasons, though, you probably deserve it. I know very little about BGP operations; I did not know that there was PKI and route validation like they described in the article.
They'd probably be less aggressive if they'd been able to reach anyone there or had received any response (none as of 8 hours after the incident). As noted above in this discussion the CF team thought they had appropriate contact information for all top tier carriers, and I suspect they do have what Verizon would call the appropriate contact info. Not much they can do if Verizon ghosts them, though.
I guess they could take steps to null route everything to/from Verizon to see if they could get someone's attention that way.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#64Would this have affected stuff in the UK? All sorts of sites like MealPal were inaccessible this morning for a bit
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#65Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#66Every time I see the BGP abbreviation it's about a huge fuck-up. Either somebody hijacks routes intentionally or something like this happens.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#67Earlier quoted context omitted.
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…
The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.
I can't seem to muster much sympathy for any publicly-traded US ISP not performing technical due diligence.
If they can afford to lobby against non-profit competition and for local monopolies, they should damn well be able to staff a NOC for this type of issue.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#68Verizon's lucky it's a blog post that doesn't mince words, rather than a lawsuit.
Can they get a lawsuit?. Has Verizon broken their SLA?. Is there a manual to mitigate all the edge cases? What about being aware internally this had to be improved but it was delayed due bureaucracy.
Torturous influence or something maybe?
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#69Every time I see the BGP abbreviation it's about a huge fuck-up. Either somebody hijacks routes intentionally or something like this happens.
As any other critical underlying infrastructure of our lives, it's taken for granted and ignored until it breaks.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#70Would this have affected stuff in the UK? All sorts of sites like MealPal were inaccessible this morning for a bit
I wouldn't be surprised if various other transit providers slurped the bogus routes up from Verizon (without filtering), so it's certainly possible.