Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

181–190 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#181
post #120
post #85

Earlier quoted context omitted.

So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.

Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.

Wow I've got some beachfront property in Nevada to sell you if you think the "web of trust" actually addressed any credible threat model.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#182

Earlier quoted context omitted.

Cloudflare's bet is essentially that they can control so much of the internet infrastructure that they can behave however they like and we all simply have to deal with it.

> behave however they like and we all simply have to deal with it So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??

Not really. You don't have to be a massive player to screw things up with BGP.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#183

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

Very nice writeup on RPKI! I don't know anything about network engineering, but it appears that RPKI will distribute trust from ISPs to RIRs (Regional Internet Registries) like ARIN and RIPE. As I understand it, the RIR will sign your IP allocation with RPKI, which means fat-fingering on your side will result in the ISP not finding you as it takes BGP announcement and RIR confirmation for the ISP to acknowledge your…

RPKI uses CAs at the RIRs because the RIRs are who make the IP allocations and have a relationship with the IP holders and can (at least in theory) authenticate the holders.

Just as a RIR could issue a certificate for your IPs to someone else, they could change WHOIS, which is how IP delegations are generally cross referenced.

You're welcome to accept (or propagate) someone's advertisements without RPKI in case of some dispute with their RIR, but expect to get called out for it if the routes are bogus if you don't answer your NOC phone or email or twitters.

Actually, I don't think Cloudflare was even calling Verizon out for not doing RPKI, which is fairly new and has costs, it was more for not limiting prefix counts; a small customer should probably be limited to 2n + 4 prefixes where N is the average number of prefixes they've advertised over the past 30 days; or like they have to put their prefixes in a portal or something.

Filtering customer advertisements with IRRs is also pretty normal.

But really, you gotta answer the phone. The steel guys answered the phone.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#184
post #120

Earlier quoted context omitted.

Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.

Other than "not enough people are interested" what is stopping you or any group of people from using such a decentralized system as your primary name resolver today? I.e. if it's not in the web of trust use existing DNS as a fallback and watch it grow. I'm not sure I'd trust such a system to prevent banksite.com from being hijacked but I don't need to for you to.

Needs more blockchain

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#185

Earlier quoted context omitted.

> behave however they like and we all simply have to deal with it So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??

Not really. You don't have to be a massive player to screw things up with BGP.

No doubt.

But it’s 2019 and I can’t muster up much sympathy for a tier 1 who can’t get inbound filters and a responsive NOC implemented correctly - things which were table stakes in 2009.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#186
post #85
post #60

Earlier quoted context omitted.

Hoisting my pitchfork a bit, but the internet might be better off without hierarchical DNS. I certainly wouldn't call that "the world ending."

So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.

Content-addressable schemes seem to be pretty effective in their respective niches. You lose the semantic component of dns, though. Perhaps you could add some sort of local name pinning.

If we imagine the internet is going to keep expanding at anywhere near its historical rate it seems like we might have to let go of the idea of letting a single entity universally control a namespace.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#187

Earlier quoted context omitted.

I do love that the CEO of Cloudflare is throwing technical shade at Verizon and others here and on Twitter for being useless.

Do you really think that "throwing shade" is what the internet needs? Is "throwing shade" an admirable quality in someone who is supposed to be demonstrating leadership? Anyone who has worked as a network engineer for a major ISP knows the internet is quite brittle. During my entire time in that profession I can't remember a time when attempting to shame people was used to resolve a routing issue or to improve relati…

Shame is one of the most effective tools in influencing human behavior, and from the sounds of this post and the other coverage on the incident, Verizon has earned far more ire than is directed at them in this blog post.

A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course; if you mess up the fonts in the end of month report to your super and he calls you a fucking idiot, he's probably an unbalanced person in need of mental help. If on the other hand you knock dead 15% of GLOBAL Internet traffic out of sheer laziness, I'd say you've earned more than a few 'go fuck yourself's.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#188

Earlier quoted context omitted.

I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…

>Did Cloudflare do that? Yes? >Cloudflare has decided that it's high-time we took a leadership role to finally secure BGP routing etc. https://blog.cloudflare.com/rpki/ >their CEO is on Twitter telling Verizon they should be ashamed Yes, well >I'll be the first to line up for a good publish lashing of US ISPs

There's a huge difference between saying you're going to be a leader and, y'know, actually being a leader. And there's an even huger difference between that and being an effective leader. I follow Cloudflare and eastdakota a lot. He clearly has the capability to be an effective leader (he is a CEO after all), and I personally admire him. However, in this particular situation, publicly berating the people that he is supposedly taking a "leadership role" over does not a good leader make.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#189

Earlier quoted context omitted.

I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…

To be frank, your post makes it clear that you don't know the context. CF simply cannot do anything on their own to mitigate the problem where Verizon constructs bad BGP routes to Cloudflare IPs and then advertises those routes to third parties. The only mitigation possible is to contact whoever's advertising the bad routes and get them to stop.

Have you read Cloudflare's multiple blog posts regarding BGP? Did you read the tweets from their directors talking about how other customers were unaffected by the event today because of the mitigations put in place? Did you even do the simplest Google about BGP protocols and the plans in place to prevent this from happening in the future?

If you're going to try to impose yourself as the gatekeeper of "knowing the context", you should probably know it yourself. Saying CF "simply cannot do anything" is narrow minded at best, and completely wrong otherwise. In fact, in this very blog post linked in the OP, Cloudflare talks about taking steps to mitigate BGP issues in the future. That's great, if only it wasn't also paired with a childish finger pointing session.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#190
post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

Yes, very much professionalism including such recent email threads titled "Russian Anal Probing"
Post reply on HN