Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

171–180 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#171

Post by harry8 »

floatingatoll wrote: post #164
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

There’s no excuse for the personal slander here - “his beauty rest interrupted” is a mean-spirited, inappropriate way to discuss Cloudflare’s post. Please don’t insult people unnecessarily here.

I read it as a reasonable and slightly humorous way to acknowledge that the author is a human being and may have been (justifiably?) in a bad mood writing the post because lack of sleep does that. No idea of how sensible that suggestion is or isn't in this case but didn't really think it was derogatory. A human being is not a machine. Sanity-sleep might be a better term for it but nobody uses that whereas beauty-sleep is in common, albeit ironic, usage.

Let's all take our sleep seriously, yeah? :-)

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#172

Post by ilaksh »

eastdakota wrote: post #40
yingw787 wrote: post #23

Earlier quoted context omitted.

Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

It's critical for the internet to work. Actually, some types of emergency services rely on it.

So this level of negligence is dangerous. Shouldn't there be criminal charges? Or at least some kind of legal action.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#174

Post by xen2xen1 »

OBLIQUE_PILLAR wrote: post #94

I hope we remember this thread the next time Shady Russians or Crafty Chinese make a BGP mistake and are accused of Active Measures.

This is an obvious mistake made by a steel company. The BGP "errors" that make tons of traffic go to Russia or China OFTEN WORK COMPLETELY. Some have worked so well no one noticed for months. That's far, far different than an error, that a large section of a country's traffic being successfully hijacked to another country. Accidents usually break things. What Russia and China are too successful to be accidents.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#175

Post by dogecoinbase »

bogomipz wrote: post #170
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

Cloudflare's bet is essentially that they can control so much of the internet infrastructure that they can behave however they like and we all simply have to deal with it.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#176

Post by bogomipz »

eastdakota wrote: post #40
yingw787 wrote: post #23

Earlier quoted context omitted.

Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

Your comments indicate to me that you don't really understand RPKI beyond a marketing value because if you did you wouldn't be throwing it around as some silver bullet. RPKI although a step in the right direction is also susceptible to misconfiguration and attack by hostile entities[1]. Additionally outages as bad as any BGP misconfiguration are also possible if an RIR's RPKI repo becomes unavailable. This has already happened. See:

https://www.arin.net/vault/announcements/2018/20181024_updat...

and

https://www.ripe.net/support/service-announcements/service-a...

There are also issues such a broken legacy ROAS: https://blog.apnic.net/2018/10/16/cleaning-up-roas-inconsist...

And the list goes on. Please stop with the hype and hand waving.

[1] https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#177

Post by bogomipz »

MrStonedOne wrote: post #147
bogomipz wrote: post #126

From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…

It was gross negligence that leads to this. Nothing more.

Do you work for Verizon's NOC or Network Engineering department then? You have inside knowledge that it was negligence? Because I provided a specific scenario where it would not be "gross negligence."

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#178

Post by owenmarshall »

dogecoinbase wrote: post #175
bogomipz wrote: post #170

Earlier quoted context omitted.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

Cloudflare's bet is essentially that they can control so much of the internet infrastructure that they can behave however they like and we all simply have to deal with it.

> behave however they like and we all simply have to deal with it

So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#179

Post by btown »

anonbutler wrote: post #152
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

Tom is based in London. So he had a good night's sleep and was well rested.

We don’t know that he wasn’t up late fixing another bug that we thankfully never saw, and that his life hasn’t been like a season of 24 this past day.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#180

Post by bogomipz »

sofaofthedamned wrote: post #91
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

I do love that the CEO of Cloudflare is throwing technical shade at Verizon and others here and on Twitter for being useless.

Do you really think that "throwing shade" is what the internet needs? Is "throwing shade" an admirable quality in someone who is supposed to be demonstrating leadership? Anyone who has worked as a network engineer for a major ISP knows the internet is quite brittle. During my entire time in that profession I can't remember a time when attempting to shame people was used to resolve a routing issue or to improve relations in order to resolve future routing issues.
Post reply on HN