I remember the early days of the Internet, when I could log in to an ISP router running BGP, with a blazing fast T1 to an early tier 1 Internet provider. We could literally announce any route we wanted, no filtering. We used to regularly black hole spammers, then turn them back on an hour or two later.
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
21–30 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#22I still get a Cloudflare 1020 error here: http://shadow.tech (my location : Scandinavia ) Are these sites waiting for some kind of propagation or cache busting? It's a pretty large gaming service.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#23Thank you for the summary. And, a sincere thank you for not mincing words when it comes to something as important as this. > However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare. > IRR filtering would not have increased Verizon's costs or limited their service in any way…
Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#24I am surprised that CF is as aggressive toward Verizon in public as they are. Once you start breaking the Internet for stupid reasons, though, you probably deserve it. I know very little about BGP operations; I did not know that there was PKI and route validation like they described in the article.
... Sometimes you just have to call a spade a spade. I know a little about BGP, having run it for a small provider in the distant past, and Verizon should have never been in a position to cause this.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#25Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2…
[deleted]
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#26> For example, our own IPv4 route 104.20.0.0/20 was turned into 104.20.0.0/21 and 104.20.8.0/21. [...] The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is. Did RPKI help reduce the scope of this incident, by stopping propagation of these faulty routes earlier than otherwise? Or did it have no effect in t…
Anecdotal, but:
The article notes that AT&T has implemented RPKI, and a client mentioned to me that he wasn't having problems accessing Cloudflare-hosted infrastructure via his AT&T phone. The rest of his employees were having major issues though via the municipal fiber service provider.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#27> All of the above suggestions are nicely condensed into MANRS (Mutually Agreed Norms for Routing Security)
Whoever came up with that name and acronym deserves an award.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#28Is Verizon's engineer on vacation, or did he just call in sick today?
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#29Verizon's lucky it's a blog post that doesn't mince words, rather than a lawsuit.
Can they get a lawsuit?. Has Verizon broken their SLA?. Is there a manual to mitigate all the edge cases? What about being aware internally this had to be improved but it was delayed due bureaucracy.