Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

151–160 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#151

Post by henryfjordan »

MrStonedOne wrote: post #114
clvx wrote: post #29

Earlier quoted context omitted.

Can they get a lawsuit?. Has Verizon broken their SLA?. Is there a manual to mitigate all the edge cases? What about being aware internally this had to be improved but it was delayed due bureaucracy.

>edge cases? This is not an edge case, allowing downstream networks to broadcast routes for networks they do not own is a very well known security and operational issue with operating an ISP. Massive parts of the internet went down in the 90s to teach us this lesson. Likewise, bureaucracy does not excuse not fixing an issue thats existed since the 90s, and not deploying any 1 of 3 mitigation tricks (let alone all 3).…

Who has standing though?

A Verizon customer might say "my internet was down" but there is 100% a clause in their contract about outages and SLAs.

Any company that lost sales likely doesn't have a contract with them, so what are they going to sue for? "Verizon didn't carry my 1s and 0s for free this morning"? Person A on the freeway having an accident and causing Person B to sit in traffic and miss their sales meeting isn't liable for that...

Maybe their peers (other telcos) have more standing because they couldn't deliver to their customers as a result but they of course all have a clause in their contracts about outages and SLAs that means ultimately they lost no money so there are no damages.

And this is why we need government regulation, either to break up the Telcos or nationalize them

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#152

Post by anonbutler »

jawns wrote: post #146
londons_explore wrote: post #103

Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours . I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a water…

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

Tom is based in London. So he had a good night's sleep and was well rested.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#153

Post by henryfjordan »

MrStonedOne wrote: post #117
azernik wrote: post #90

Earlier quoted context omitted.

Nope - because said collective action would probably involve denying service to tens of millions of Verizon customers.

They could do better

Can they? How many choices of ISP do you have? I have 2, Spectrum (Time Warner) or some no-name for the same price with 1/10th the speed. I could not practically switch off Time Warner if I wanted to without moving. I imagine that Verizon customers are in the same boat.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#154

Post by SpicyLemonZest »

txcwpalpha wrote: post #136
SpicyLemonZest wrote: post #129

Earlier quoted context omitted.

The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that: 1. They implement basic precautions to prevent dumb things from going wrong. 2. They're available 24/7, to immediately respond to and remediate whatever does go wrong. 3. Both of the above…

I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…

To be frank, your post makes it clear that you don't know the context. CF simply cannot do anything on their own to mitigate the problem where Verizon constructs bad BGP routes to Cloudflare IPs and then advertises those routes to third parties. The only mitigation possible is to contact whoever's advertising the bad routes and get them to stop.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#155

Post by jrockway »

rattray wrote: post #148
brians wrote: post #133

Earlier quoted context omitted.

(deleted) Yeah, that wasn’t contributing. Everyone has bad days.

Your profile mentions: > Chief Architect, Information Security, Akamai Technologies. I do not speak for my employer. Probably best to disclose this more directly in comments on topics related to competitors.

Why? Comments don't typically disclose "I am a corporate spambot" and yet such things exist, so you should probably read every comment as though it has some angle / vested interest. At least brians was nice enough to mention their bias in their profile. Most cases won't be so easy. Making people "opt in" to disclosing biases will just make it easier for the real bad actors to slip through.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#156

Post by jrockway »

w8vY7ER wrote: post #109
tssva wrote: post #86

AS701 was the UUNET/Worldcom AS for the US and eventually the US/Canada network. Verizon bought Worldcom in 2006 and they became Verizon Business. From the late 90s to early 2000s I worked for UUNET/Worldcom as an engineer in the network planning and design group. I worked in the international group but among other things we were responsible for the build out of AS701 into Canada, the exchange sites where AS701 conne…

As they should, this is day-one BGP configuration stuff. Nothing advanced, nothing especially technical or time consuming. Speaks to a gross degree of professional negligence on the part of AS701. Really disappointing to see this degree of ignorance.

Networking infrastructure / tools are in the stone age compared to what we have in the software world. I am inclined to cut them some slack.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#157

Post by [deleted] »

inopinatus wrote: post #104
eropple wrote: post #4

Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2…

Back in that oft-forgotten age I was privileged to know, work, and chill with the brave volunteers (and, subsequently, paid RIR staff) holding down the buttons. It’s worth mentioning that even back then there dwelt in the west a large ugly troll whose name was AS701 (AS701 was not alone, either, having two hideous siblings, AS702 and AS703, that lived in other climes). Everyone tiptoed around the beast, because when…

[deleted]

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#158

Post by mkl »

txcwpalpha wrote: post #110
psim1 wrote: post #57

Earlier quoted context omitted.

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional. What strikes me the most is that this whole "event" woul…

I, in New Zealand, was well aware due to the number of inaccessible websites. It was a significant event not just on many people's radar, but actively preventing them from doing everyday things.

That said, I do think the tone of this blog post may have been taken a bit far.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#159

Post by sgentle »

psim1 wrote: post #57
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

I think your definition of unprofessional needs recalibrating if it applies more to the people calling out professional negligence than to the people committing it.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#160

Post by robbiemitchell »

brians wrote: post #133
cortesoft wrote: post #123

Earlier quoted context omitted.

Blog posts are their speciality

(deleted) Yeah, that wasn’t contributing. Everyone has bad days.

Strong words for an outage that affected "Cloudflare, Amazon, Linode, Google, Facebook, and others"
Post reply on HN