It's clear that Verizon broke the Internet this morning through incompetent BGP management, and they could do it again. Who holds them accountable?
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
11–20 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#12Verizon's lucky it's a blog post that doesn't mince words, rather than a lawsuit.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#13One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.
We thought we did. And tried both public and private lines of communication — without reply. Still waiting.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#14It's clear that Verizon broke the Internet this morning through incompetent BGP management, and they could do it again. Who holds them accountable?
glances over at Ajit Pai
Nobody.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#15Amusing callout to pager duty in the screenshot of the call-log :)
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#16Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#17> One of our network engineers made contact with DQE Communications quickly and after a little delay they were able to put us in contact with someone who could fix the problem. DQE worked with us on the phone to stop advertising these “optimized” routes to Allegheny Technologies Inc. We're grateful for their help. Once this was done, the Internet stabilized, and things went back to normal.
It's funny how we have to still use phone to help fixing some internet routing problem, even if phone doesn't literally means the old black curly-wire equipment
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#18I appreciate how sympathetic Cloudflare is to the root-cause party because they answered the phone and undid what they shouldn’t have done.
(If my understanding is correct, they shouldn’t have told Verizon about the better routing, while Verizon should have known better)
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#19Very interesting and easy to understand overview, thanks. I'm not familiar with this side of networking, but it sounds to me the "BGP Optimiser" product was left largely to its own devices and automated a configuration change without any explicit approval from a human operator (I could be wrong) With the protocol being prone to problems like leaky routes and sloppy peers accepting them, is it really wise to leave the…
I think the reality of it is that these BGP optimizers really can't be human checked. There's just too much it is doing, and for them to be really beneficial they need to respond quickly to network path congestion. I would be surprised if overseeing such a system could be done with fewer than 6 full time people, as a WAG.
... Which is why you should be really sure that these optimized routes never leak! And on top of it, Verizon should never have accepted those announcements.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#20> For example, our own IPv4 route 104.20.0.0/20 was turned into 104.20.0.0/21 and 104.20.8.0/21. [...] The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is.
Did RPKI help reduce the scope of this incident, by stopping propagation of these faulty routes earlier than otherwise? Or did it have no effect in this case?