Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

41–50 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#41
post #14

Earlier quoted context omitted.

glances over at Ajit Pai Nobody.

It doesn't need government intervention. It needs other companies to hold them accountable.

Absolutely. If you are a Verizon service provider customer, please call them and register your feelings on this matter. Make sure you let them know in no uncertain terms that you are considering switching providers based on their lack of following best practices.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#42

Earlier quoted context omitted.

Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

Are you able to comment on how a company like yourself, or the other companies which were affected, can pursue anything with Verizon? Or is Verizon free to continue with bad practices and have a repeat of this issue?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#43
post #4

Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2…

You came up with this analogy when Lost was still airing, no?

Although BGP probably makes about the same amount of sense to most people.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#44
post #35

Earlier quoted context omitted.

Works from the ATL DC, what is the airport code that shows up on https://cloudflare-test.judge.sh/#shadow.tech ? Might be a local [maybe routing] issue with CF -> shadow's web server.

Not working here either (Finland), that page shows HEL for me. shadow.tech shows "Error 1020 Ray ID: 4ec1c24b2a945b25 • 2019-06-24 21:22:45 UTC Access denied What happened? This website is using a security service to protect itself from online attacks."

Oh, 1020 access denied means some firewall rule (any combination of rules [0] and `if`/`or` statements) or access controls (such as blocking IP ranges and countries) blocked access to the website. These are always set up by the site operator, so this isn't caused by any CF issues or the earlier routing issues.

I guess best course of action for those who want to access the site is to tweet them https://twitter.com/shadow_official with your Ray ID.

0: https://developers.cloudflare.com/firewall/cf-firewall-rules...

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#45

Earlier quoted context omitted.

It doesn't need government intervention. It needs other companies to hold them accountable.

Absolutely. If you are a Verizon service provider customer, please call them and register your feelings on this matter. Make sure you let them know in no uncertain terms that you are considering switching providers based on their lack of following best practices.

I think that may be a little too subtle for the parent, but well done.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#46
Over a decade ago one of my friends was banned from the Sheffield Uni network for playing around with BGP and knocking the whole campus offline. One kind of has to wonder whether Verizon can suffer the same consequences simply by collective action on the part of other affected parties.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#48
post #29
post #12

Verizon's lucky it's a blog post that doesn't mince words, rather than a lawsuit.

Can they get a lawsuit?. Has Verizon broken their SLA?. Is there a manual to mitigate all the edge cases? What about being aware internally this had to be improved but it was delayed due bureaucracy.

Anybody can sue. And it'll still cost Verizon real money to fight those suits, even if they truly don't have liability.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#49
post #35

Earlier quoted context omitted.

Works from the ATL DC, what is the airport code that shows up on https://cloudflare-test.judge.sh/#shadow.tech ? Might be a local [maybe routing] issue with CF -> shadow's web server.

Not working here either (Finland), that page shows HEL for me. shadow.tech shows "Error 1020 Ray ID: 4ec1c24b2a945b25 • 2019-06-24 21:22:45 UTC Access denied What happened? This website is using a security service to protect itself from online attacks."

[deleted]

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#50
post #10

One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.

We thought we did. And tried both public and private lines of communication — without reply. Still waiting.

So what's the over/under on Verizon actually picking up?
Post reply on HN