Earlier quoted context omitted.
So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.
Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
181–190 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#182Earlier quoted context omitted.
Cloudflare's bet is essentially that they can control so much of the internet infrastructure that they can behave however they like and we all simply have to deal with it.
> behave however they like and we all simply have to deal with it So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#183Earlier quoted context omitted.
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…
Very nice writeup on RPKI! I don't know anything about network engineering, but it appears that RPKI will distribute trust from ISPs to RIRs (Regional Internet Registries) like ARIN and RIPE. As I understand it, the RIR will sign your IP allocation with RPKI, which means fat-fingering on your side will result in the ISP not finding you as it takes BGP announcement and RIR confirmation for the ISP to acknowledge your…
Just as a RIR could issue a certificate for your IPs to someone else, they could change WHOIS, which is how IP delegations are generally cross referenced.
You're welcome to accept (or propagate) someone's advertisements without RPKI in case of some dispute with their RIR, but expect to get called out for it if the routes are bogus if you don't answer your NOC phone or email or twitters.
Actually, I don't think Cloudflare was even calling Verizon out for not doing RPKI, which is fairly new and has costs, it was more for not limiting prefix counts; a small customer should probably be limited to 2n + 4 prefixes where N is the average number of prefixes they've advertised over the past 30 days; or like they have to put their prefixes in a portal or something.
Filtering customer advertisements with IRRs is also pretty normal.
But really, you gotta answer the phone. The steel guys answered the phone.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#184Earlier quoted context omitted.
Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.
Other than "not enough people are interested" what is stopping you or any group of people from using such a decentralized system as your primary name resolver today? I.e. if it's not in the web of trust use existing DNS as a fallback and watch it grow. I'm not sure I'd trust such a system to prevent banksite.com from being hijacked but I don't need to for you to.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#185Earlier quoted context omitted.
> behave however they like and we all simply have to deal with it So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??
Not really. You don't have to be a massive player to screw things up with BGP.
But it’s 2019 and I can’t muster up much sympathy for a tier 1 who can’t get inbound filters and a responsive NOC implemented correctly - things which were table stakes in 2009.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#186Earlier quoted context omitted.
Hoisting my pitchfork a bit, but the internet might be better off without hierarchical DNS. I certainly wouldn't call that "the world ending."
So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.
If we imagine the internet is going to keep expanding at anywhere near its historical rate it seems like we might have to let go of the idea of letting a single entity universally control a namespace.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#187Earlier quoted context omitted.
I do love that the CEO of Cloudflare is throwing technical shade at Verizon and others here and on Twitter for being useless.
Do you really think that "throwing shade" is what the internet needs? Is "throwing shade" an admirable quality in someone who is supposed to be demonstrating leadership? Anyone who has worked as a network engineer for a major ISP knows the internet is quite brittle. During my entire time in that profession I can't remember a time when attempting to shame people was used to resolve a routing issue or to improve relati…
A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course; if you mess up the fonts in the end of month report to your super and he calls you a fucking idiot, he's probably an unbalanced person in need of mental help. If on the other hand you knock dead 15% of GLOBAL Internet traffic out of sheer laziness, I'd say you've earned more than a few 'go fuck yourself's.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#188Earlier quoted context omitted.
I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…
>Did Cloudflare do that? Yes? >Cloudflare has decided that it's high-time we took a leadership role to finally secure BGP routing etc. https://blog.cloudflare.com/rpki/ >their CEO is on Twitter telling Verizon they should be ashamed Yes, well >I'll be the first to line up for a good publish lashing of US ISPs
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#189Earlier quoted context omitted.
I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…
To be frank, your post makes it clear that you don't know the context. CF simply cannot do anything on their own to mitigate the problem where Verizon constructs bad BGP routes to Cloudflare IPs and then advertises those routes to third parties. The only mitigation possible is to contact whoever's advertising the bad routes and get them to stop.
If you're going to try to impose yourself as the gatekeeper of "knowing the context", you should probably know it yourself. Saying CF "simply cannot do anything" is narrow minded at best, and completely wrong otherwise. In fact, in this very blog post linked in the OP, Cloudflare talks about taking steps to mitigate BGP issues in the future. That's great, if only it wasn't also paired with a childish finger pointing session.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#190Earlier quoted context omitted.
Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…