Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#111

Post by empath75 »

My favorite outage when I worked for a voip company was when one of our tech support people told a new customer that she needed to ‘add our ip address to your router’, meaning add it to the firewall whitelist, but she repeated that verbatim to the telco tech who misunderstood and then escalated her way up the chain at a major telco until some engineer with the wrong rights said ‘fuck it’ and updated bgp to route all of our traffic down her T-1 line.

That was a fun conference call, and listening to the lady on the phone I could see how the engineer got to that point.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#112

Post by throw7 »

ziddoap wrote: post #6

Thank you for the summary. And, a sincere thank you for not mincing words when it comes to something as important as this. > However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare. > IRR filtering would not have increased Verizon's costs or limited their service in any way…

Back in the suspender wearing neckbeards days, the answer was simple... blackhole all Verizon routes.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#113

Post by MBCook »

teraflop wrote: post #95
MBCook wrote: post #71

Can someone explain why the optimizer would split one route into two? Wouldn’t it be more optimized to coalesce routes whenever possible?

If an ISP has multiple physical connections that it could use to reach Cloudflare's network, it makes sense to distribute the traffic that's addressed to different IPs across different links, instead of using a single route that sends all the traffic over one link and leaves the others idle.

Good point. Thanks.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#114

Post by MrStonedOne »

clvx wrote: post #29
camgunz wrote: post #12

Verizon's lucky it's a blog post that doesn't mince words, rather than a lawsuit.

Can they get a lawsuit?. Has Verizon broken their SLA?. Is there a manual to mitigate all the edge cases? What about being aware internally this had to be improved but it was delayed due bureaucracy.

>edge cases?

This is not an edge case, allowing downstream networks to broadcast routes for networks they do not own is a very well known security and operational issue with operating an ISP. Massive parts of the internet went down in the 90s to teach us this lesson.

Likewise, bureaucracy does not excuse not fixing an issue thats existed since the 90s, and not deploying any 1 of 3 mitigation tricks (let alone all 3).

Negligence causing damage from lost sales/traffic is sue-able.

The case would basically resolve around rather or not V had an obligation to prevent this from happening, and rather or not they were grossly negligent in that obligation.

In my view the answer is yes.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#115

Post by kortilla »

eastdakota wrote: post #40
yingw787 wrote: post #23

Earlier quoted context omitted.

Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

Note that RPKI won’t prevent outages caused by route leaks (because the leak has a valid signed origin).

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#116

Post by codebje »

yingw787 wrote: post #80
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

Very nice writeup on RPKI! I don't know anything about network engineering, but it appears that RPKI will distribute trust from ISPs to RIRs (Regional Internet Registries) like ARIN and RIPE. As I understand it, the RIR will sign your IP allocation with RPKI, which means fat-fingering on your side will result in the ISP not finding you as it takes BGP announcement and RIR confirmation for the ISP to acknowledge your…

The IRR is also controlled by a few entities that would be vulnerable to government intervention, but that's the tool we currently rely on.

RPKI roots trust at the RIRs, and that is a vulnerability, but any government intervention would end that trust and end the use of the RIRs as trust anchors. It's pretty unlikely to ever be used that way.

Disclaimer: I co-authored some of the drafts for RPKI and helped implement RPKI systems at an RIR.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#117

Post by MrStonedOne »

azernik wrote: post #90
hyperion2010 wrote: post #46

Over a decade ago one of my friends was banned from the Sheffield Uni network for playing around with BGP and knocking the whole campus offline. One kind of has to wonder whether Verizon can suffer the same consequences simply by collective action on the part of other affected parties.

Nope - because said collective action would probably involve denying service to tens of millions of Verizon customers.

They could do better

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#118

Post by mffnbs »

munificent wrote: post #27

> All of the above suggestions are nicely condensed into MANRS (Mutually Agreed Norms for Routing Security) Whoever came up with that name and acronym deserves an award.

Wow, thanks for pointing that out, I missed it on my once-over.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#119

Post by coldreactor »

teejmya wrote: post #77
jboy55 wrote: post #36

Here's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.

Amen. We need a support group. "Hi, I'm Teejmya, and I was on call last night"

"I'm Coldreactor and I was traumatized by the massive number of calls last night"

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#120

Post by ethbro »

bin0 wrote: post #85
ethbro wrote: post #60

Earlier quoted context omitted.

Hoisting my pitchfork a bit, but the internet might be better off without hierarchical DNS. I certainly wouldn't call that "the world ending."

So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.

Freedom isn't free. Web of trust.

Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.

Post reply on HN