[deleted]
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
31–40 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#32[deleted]
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#33I still get a Cloudflare 1020 error here: http://shadow.tech (my location : Scandinavia ) Are these sites waiting for some kind of propagation or cache busting? It's a pretty large gaming service.
Works from the ATL DC, what is the airport code that shows up on https://cloudflare-test.judge.sh/#shadow.tech ? Might be a local [maybe routing] issue with CF -> shadow's web server.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#34> For example, our own IPv4 route 104.20.0.0/20 was turned into 104.20.0.0/21 and 104.20.8.0/21. [...] The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is. Did RPKI help reduce the scope of this incident, by stopping propagation of these faulty routes earlier than otherwise? Or did it have no effect in t…
It didn't. Virtually no major operator rejects invalids. However some do more-or-less strict prefix+ASN filtering.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#35I still get a Cloudflare 1020 error here: http://shadow.tech (my location : Scandinavia ) Are these sites waiting for some kind of propagation or cache busting? It's a pretty large gaming service.
Works from the ATL DC, what is the airport code that shows up on https://cloudflare-test.judge.sh/#shadow.tech ? Might be a local [maybe routing] issue with CF -> shadow's web server.
Not working here either (Finland), that page shows HEL for me.
shadow.tech shows "Error 1020 Ray ID: 4ec1c24b2a945b25 • 2019-06-24 21:22:45 UTC Access denied What happened? This website is using a security service to protect itself from online attacks."
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#36Here's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#37One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.
We thought we did. And tried both public and private lines of communication — without reply. Still waiting.
Thank you all at Cloudflare for your attention to detail. I'm currently rethinking my stance of years believing you were crimeflare. I do hope Verizon responds and everyone can learn from this.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#38Amusing callout to pager duty in the screenshot of the call-log :)
Which is more interesting, philosophically? The internet had a problem due to a single issue, or that the internet's problem was fixed due to a single person calling various people on a cell phone?
(device interactions versus human interactions..)
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#39Earlier quoted context omitted.
glances over at Ajit Pai Nobody.
It doesn't need government intervention. It needs other companies to hold them accountable.
Oh geez looks like the other companies aren't holding them accountable and the government shouldn't hold them accountable so I guess they just aren't accountable for causing broad swathes of economic damage because they were lazily managing their networks.
The system works!
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#40Thank you for the summary. And, a sincere thank you for not mincing words when it comes to something as important as this. > However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare. > IRR filtering would not have increased Verizon's costs or limited their service in any way…
Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly).
We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smaller, less-connected network. The crazy thing about BGP is that any router can announce that it's responsible for a block of IP addresses and, if it's trusted enough, that's what the map of the Internet will reflect.
The long term solution is for networks to implement and enforce RPKI. AT&T, for instance, implemented RPKI and we did not see any drop in traffic to their network today.
Verizon not only didn't implement RPKI, which would be the best-of-breed approach, but also didn't do even basic route filtering. It's as if a trusted traffic cop (Verizon) overheard from a random passing motorist that the main road was closed and, as a result, directed all traffic off a pier and into the ocean.
More about RPKI if you're interested: https://blog.cloudflare.com/rpki/