Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

81–90 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#81

Post by crankylinuxuser »

ljm wrote: post #79
psim1 wrote: post #57

Earlier quoted context omitted.

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.

A better question:

If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#82

Post by youeeeeeediot »

psim1 wrote: post #57
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

This problem has been known for decades. As of April 2019, 56.1% of the world's population has internet access. Do you think it is acceptable for a major transit ISP to have no basic filters in 2019 let alone implement RPKI?

Can you explain what part of the Cloudflare statement you consider to be posturing? A cursory review of the BGP announcements referenced in the article are pretty clear. Facts are facts regardless of how the message is delivered.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#83

Post by ikiris »

psim1 wrote: post #57
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

What the hell do you think cloudflare was supposed to have done here?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#84

Post by nodesocket »

I saw a few of my static sites that are hosted on Cloudflare and few other auxiliary 3rd party services I use flapping back and forth on PagerDuty, but not all of my Cloudflare sites triggered down.

Is this just because the unaffected Cloudflare sites were not within the CIDR range affected?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#85

Post by bin0 »

ethbro wrote: post #60
csours wrote: post #52

Earlier quoted context omitted.

https://www.icann.org/news/blog/the-problem-with-the-seven-k...

Hoisting my pitchfork a bit, but the internet might be better off without hierarchical DNS. I certainly wouldn't call that "the world ending."

So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#86

Post by tssva »

AS701 was the UUNET/Worldcom AS for the US and eventually the US/Canada network. Verizon bought Worldcom in 2006 and they became Verizon Business.

From the late 90s to early 2000s I worked for UUNET/Worldcom as an engineer in the network planning and design group. I worked in the international group but among other things we were responsible for the build out of AS701 into Canada, the exchange sites where AS701 connected to the various other international UUNET AS's and the PoP's where dedicated circuits for international customers who wished to connect directly to AS701 would be terminated. The point being that I am familiar with how AS701 was operated at that time.

UUNET's reputation at the time might not have been sterling due to the business decision to basically be a safe haven for spammers but from a technical standpoint the network was operated at a high standard. The basic BGP filtering referenced in the article was certainly in place at the time and if this had happened then heads would have rolled.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#87

Post by eastdakota »

teejmya wrote: post #77
jboy55 wrote: post #36

Here's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.

Amen. We need a support group. "Hi, I'm Teejmya, and I was on call last night"

Yes, apologies and thank you!

If you email me (matthewatcloudflaredotcom) your shirt size, preference for men's or women's cut, and your postal address with the subject line:

"Verizon BGP Leak On Call Support Group"

I'll send you a Cloudflare tshirt. Least we can do.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#88

Post by ikiris »

kevinreedy wrote: post #62

> The RPKI framework that we implemented and deployed globally last year is designed to prevent this type of leak. It enables filtering on origin network and prefix size. The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is. Does RPKI prevent Cloudflare from announcing additional /22 routes during an inci…

And you believe the internet optimizer wouldn't have added /23s and /24s.... why?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#89

Post by laughinghan »

psim1 wrote: post #57
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

I was thinking the exact same thing—right up until I got the part where they still haven't responded 8 hours later (to say nothing of apologizing), and played no role in fixing the problem (DQE did that, apparently).

We all make mistakes. It's unreasonable to expect 100% uptime from anyone. But if you operate a service that so many people are relying on, and you make billions of dollars in profit each year (we're not talking about an unpaid volunteer open-source maintainer here), you absolutely have a responsibility to at least try to help fix it when there's a problem. It's brazenly irresponsible to go radio silent while your customer's other vendor fixes the problem.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#90

Post by azernik »

hyperion2010 wrote: post #46

Over a decade ago one of my friends was banned from the Sheffield Uni network for playing around with BGP and knocking the whole campus offline. One kind of has to wonder whether Verizon can suffer the same consequences simply by collective action on the part of other affected parties.

Nope - because said collective action would probably involve denying service to tens of millions of Verizon customers.
Post reply on HN