Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
blog.cloudflare.com
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
1–10 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#2https://news.ycombinator.com/item?id=20262214 is the earlier thread on this.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#3https://news.ycombinator.com/item?id=20262214 is the earlier thread on this.
Thanks. And thanks for the help today updating the title on the original post as the real cause came to light!
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#4Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad.
Today in 2019, “the world” is defined as “you know, the world“, and there are seven million buttons being held down all over the world.
If any four of them are released, the world ends.
We have made mistakes, is what I’m saying.
(I once had call to explain to nontechnical people how and why the internet is the way it is and why my ops crews tend to be full of people who are a little too calm about things being constantly on fire. This was my best crack at it.)
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#5Very interesting and easy to understand overview, thanks.
I'm not familiar with this side of networking, but it sounds to me the "BGP Optimiser" product was left largely to its own devices and automated a configuration change without any explicit approval from a human operator (I could be wrong)
With the protocol being prone to problems like leaky routes and sloppy peers accepting them, is it really wise to leave these BGP optimiser products running without some level of supervision?
EDIT: of course I guess the human operator might wave the change through too without fully appreciating the problem...
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#6Thank you for the summary.
And, a sincere thank you for not mincing words when it comes to something as important as this.
>However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare.
>IRR filtering would not have increased Verizon's costs or limited their service in any way. Again, the only explanation we can conceive of why it wasn't in place is sloppiness or laziness.
In an attempt to find any statement given by Verizon, I found that The Register was able to get this amazing statement:
"Verizon sent us the following baffling response to today's BGP cockup: "There was an intermittent disruption in internet service for some [Verizon] FiOS customers earlier this morning. Our engineers resolved the issue around 9am ET."" [1]
[1]https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconf...
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#7I am surprised that CF is as aggressive toward Verizon in public as they are. Once you start breaking the Internet for stupid reasons, though, you probably deserve it.
I know very little about BGP operations; I did not know that there was PKI and route validation like they described in the article.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#8Very interesting and easy to understand overview, thanks. I'm not familiar with this side of networking, but it sounds to me the "BGP Optimiser" product was left largely to its own devices and automated a configuration change without any explicit approval from a human operator (I could be wrong) With the protocol being prone to problems like leaky routes and sloppy peers accepting them, is it really wise to leave the…
the issue is that, as stated by the article. Things like this could be prevented by doing proper IRR filtering.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#9I am surprised that CF is as aggressive toward Verizon in public as they are. Once you start breaking the Internet for stupid reasons, though, you probably deserve it. I know very little about BGP operations; I did not know that there was PKI and route validation like they described in the article.
More details on RPKI if you're interested: https://blog.cloudflare.com/rpki/
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#10One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.