Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

211–220 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#211

Post by txcwpalpha »

ubercow13 wrote: post #206
txcwpalpha wrote: post #189

Earlier quoted context omitted.

Have you read Cloudflare's multiple blog posts regarding BGP? Did you read the tweets from their directors talking about how other customers were unaffected by the event today because of the mitigations put in place? Did you even do the simplest Google about BGP protocols and the plans in place to prevent this from happening in the future? If you're going to try to impose yourself as the gatekeeper of "knowing the co…

AT&T customers were unaffected because of mitigations put in place by AT&T that Verizon hasn't put in place. The steps you refer to in the blog post are ones that Verzion has to take.

Yes, and? Cloudflare themselves are the ones pushing their own company as "leaders" in this field, and being a "leader" does not mean "pointing fingers and trying to avoid blame whenever something bad happens". If they fancy themselves leaders regarding BGP, as said on their website, then they need to actually act like leaders.

And as I've said multiple times now, Cloudflare was in a great position here to stand themselves up as a strong leader on this topic to start working together with other companies (a la Verizon) to start to make real headway to fix the BGP problem. As other commenters have noted, the internet is entirely built on multiple organizations acting in good faith towards one another. Verizon failed to do that, and Cloudflare's response also failed to do that. I said it in another comment, but I'll also say it here: publicly berating the people that you are supposedly taking a leadership position over is not good leadership. This entire episode is going to do nothing to encourage Verizon to work closely with CF to fix this issue. In fact, I imagine it will do the exact opposite.

Today was a display of incompetence from Verizon, and a display of bad leadership by Cloudflare. I have no idea why any objective-minded person would be applauding Cloudflare for this. As I mentioned elsewhere, I would normally love a good public bashing of Verizon, but not when it comes at the cost of professionalism and progress.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#212

Post by IntelMiner »

jmvoodoo wrote: post #125
eastdakota wrote: post #13

Earlier quoted context omitted.

We thought we did. And tried both public and private lines of communication — without reply. Still waiting.

Could always announce a few of their routes and wait for them to call you I guess? :)

Arrange to drop all traffic from Verizon

Then wait just as long to pick up the phone

The sheer amount of calls taken would cost them money, the only thing they actively seem to notice

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#213

Post by takeda »

KirinDave wrote: post #181
ethbro wrote: post #120

Earlier quoted context omitted.

Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.

Wow I've got some beachfront property in Nevada to sell you if you think the "web of trust" actually addressed any credible threat model.

Not a threat model, but gives me freedom to decide whom I trust.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#214

Post by LIV2 »

saxonww wrote: post #132
Scoundreller wrote: post #18

I appreciate how sympathetic Cloudflare is to the root-cause party because they answered the phone and undid what they shouldn’t have done. (If my understanding is correct, they shouldn’t have told Verizon about the better routing, while Verizon should have known better)

I read this as Allegheny's fault, actually. DQE published to Allegheny (DQE's customer), who in turn re-published to Verizon (Allegheny's other provider). While most of the 'prevention' section talks about what Verizon didn't do, it doesn't seem to mention that Allegheny should not have re-published the DQE-published routes up to Verizon. It's startling that Verizon doesn't appear to have any leak mitigations in plac…

No customer should be able to bring down the internet due to misconfiguration this is all on Verizon imho

Any ISP worth their salt has route filtering on any customer connections, nobody should be able to announce prefixes they don't own if the ISP is doing their job properly.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#215

Post by [deleted] »

ziddoap wrote: post #42
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

Are you able to comment on how a company like yourself, or the other companies which were affected, can pursue anything with Verizon? Or is Verizon free to continue with bad practices and have a repeat of this issue?

[deleted]

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#216

Post by godzillabrennus »

adolph wrote: post #184
zamadatix wrote: post #144

Earlier quoted context omitted.

Other than "not enough people are interested" what is stopping you or any group of people from using such a decentralized system as your primary name resolver today? I.e. if it's not in the web of trust use existing DNS as a fallback and watch it grow. I'm not sure I'd trust such a system to prevent banksite.com from being hijacked but I don't need to for you to.

Needs more blockchain

https://www.namecoin.org/

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#217

Post by [deleted] »

FussyZeus wrote: post #187
bogomipz wrote: post #180

Earlier quoted context omitted.

Do you really think that "throwing shade" is what the internet needs? Is "throwing shade" an admirable quality in someone who is supposed to be demonstrating leadership? Anyone who has worked as a network engineer for a major ISP knows the internet is quite brittle. During my entire time in that profession I can't remember a time when attempting to shame people was used to resolve a routing issue or to improve relati…

Shame is one of the most effective tools in influencing human behavior, and from the sounds of this post and the other coverage on the incident, Verizon has earned far more ire than is directed at them in this blog post. A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course;…

[deleted]

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#218

Post by Dylan16807 »

txcwpalpha wrote: post #211
ubercow13 wrote: post #206

Earlier quoted context omitted.

AT&T customers were unaffected because of mitigations put in place by AT&T that Verizon hasn't put in place. The steps you refer to in the blog post are ones that Verzion has to take.

Yes, and? Cloudflare themselves are the ones pushing their own company as "leaders" in this field, and being a "leader" does not mean "pointing fingers and trying to avoid blame whenever something bad happens". If they fancy themselves leaders regarding BGP, as said on their website, then they need to actually act like leaders. And as I've said multiple times now, Cloudflare was in a great position here to stand them…

Companies respond just fine to public scrutiny, caused by them being rightfully and loudly blamed. The way you lead people isn't the same as the way you lead companies.

Verizon was acting so badly that it's clear the pure friendly approach was doing absolutely nothing. And I'm sure Cloudflare is willing to give very real and pleasant engineering help if desired.

If Verizon doesn't want to talk to Cloudflare, that's fine too. This is not a problem that requires active cooperation. They just have to do their job.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#219

Post by ikiris »

tssva wrote: post #86

AS701 was the UUNET/Worldcom AS for the US and eventually the US/Canada network. Verizon bought Worldcom in 2006 and they became Verizon Business. From the late 90s to early 2000s I worked for UUNET/Worldcom as an engineer in the network planning and design group. I worked in the international group but among other things we were responsible for the build out of AS701 into Canada, the exchange sites where AS701 conne…

I for one remember UUNET being expensive transit, but rock fucking solid.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#220

Post by Varriount »

bogomipz wrote: post #191
FussyZeus wrote: post #187

Earlier quoted context omitted.

Shame is one of the most effective tools in influencing human behavior, and from the sounds of this post and the other coverage on the incident, Verizon has earned far more ire than is directed at them in this blog post. A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course;…

How about leaking session tokens and other sensitive data for millions of people during "Cloudbleed"? Were you advocating public shaming for Cloudflare then? Was that also "sheer laziness" and did they earn "more than a few" of your "go fuck yourself's"?

I fail to see how Cloudbleed and this event are the same. Cloudbleed was caused by a Cloudflare bug, true, but it wasn't caused by outright laziness (which this incident clearly was). Furthermore, unlike Verizon's distinct lack of communication regarding this incident, Cloudflare has generally been very good about reporting and communicating with the community.
Post reply on HN