Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

91–100 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#91

Post by sofaofthedamned »

eastdakota wrote: post #40
yingw787 wrote: post #23

Earlier quoted context omitted.

Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

I do love that the CEO of Cloudflare is throwing technical shade at Verizon and others here and on Twitter for being useless.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#92

Post by eropple »

ethbro wrote: post #43
eropple wrote: post #4

Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2…

You came up with this analogy when Lost was still airing, no? Although BGP probably makes about the same amount of sense to most people.

I did not. Never saw it either. ;)

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#93

Post by laughinghan »

Scoundreller wrote: post #18

I appreciate how sympathetic Cloudflare is to the root-cause party because they answered the phone and undid what they shouldn’t have done. (If my understanding is correct, they shouldn’t have told Verizon about the better routing, while Verizon should have known better)

I don't think your understanding is correct. I think they're supposed to be allowed to "optimize" the BGP routes that they advertise to their customer (Allegheny). I'm unclear on whether Allegheny should have relayed that advertisement to Verizon, but it's clear that Verizon should definitely not have then broadcasted that to everyone else.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#95

Post by teraflop »

MBCook wrote: post #71

Can someone explain why the optimizer would split one route into two? Wouldn’t it be more optimized to coalesce routes whenever possible?

If an ISP has multiple physical connections that it could use to reach Cloudflare's network, it makes sense to distribute the traffic that's addressed to different IPs across different links, instead of using a single route that sends all the traffic over one link and leaves the others idle.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#96

Post by mschuster91 »

crankylinuxuser wrote: post #81
ljm wrote: post #79

Earlier quoted context omitted.

I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.

A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?

That's a dangerous precedent to set - because then others will jump in and demand knocking other countries (think Cuba, Venezuela, Iran, Syria or any other country on the US sanction list) offline.

IIRC the only cases where this has happened was when a couple of self-proclaimed "bulletproof hosters" were booted off of their uplinks, but even this wasn't a direct partition of the Internet.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#97

Post by ecnahc515 »

crankylinuxuser wrote: post #81
ljm wrote: post #79

Earlier quoted context omitted.

I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.

A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?

That would just result in Verizon customers being unable to access stuff which isn't good either. Their users don't have much say in the matter, and due to internet monopolies in the US, may even be the only option for some people. They literally cannot even vote with their wallet, and thus any type of repercussion to Verizon would mostly be affecting the users.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#98

Post by azernik »

woliveirajr wrote: post #17

> One of our network engineers made contact with DQE Communications quickly and after a little delay they were able to put us in contact with someone who could fix the problem. DQE worked with us on the phone to stop advertising these “optimized” routes to Allegheny Technologies Inc. We're grateful for their help. Once this was done, the Internet stabilized, and things went back to normal. It's funny how we have to s…

Gotta have a channel that's out-of-band with the internet to fix problems with the internet.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#99

Post by Tossrock »

eastdakota wrote: post #13
qaq wrote: post #10

One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.

We thought we did. And tried both public and private lines of communication — without reply. Still waiting.

I love the inclusion of the "help4u@verizon.com" address in the email attempting to get their attention on a Severity 0 event. It was worth a shot!

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#100

Post by jedberg »

tssva wrote: post #86

AS701 was the UUNET/Worldcom AS for the US and eventually the US/Canada network. Verizon bought Worldcom in 2006 and they became Verizon Business. From the late 90s to early 2000s I worked for UUNET/Worldcom as an engineer in the network planning and design group. I worked in the international group but among other things we were responsible for the build out of AS701 into Canada, the exchange sites where AS701 conne…

They most likely upgraded the equipment since the build out and never put the filtering back.
Post reply on HN