Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

101–110 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#101

Post by jedberg »

azernik wrote: post #98
woliveirajr wrote: post #17

> One of our network engineers made contact with DQE Communications quickly and after a little delay they were able to put us in contact with someone who could fix the problem. DQE worked with us on the phone to stop advertising these “optimized” routes to Allegheny Technologies Inc. We're grateful for their help. Once this was done, the Internet stabilized, and things went back to normal. It's funny how we have to s…

Gotta have a channel that's out-of-band with the internet to fix problems with the internet.

Nowadays with voip “the phone” isn’t as out of band as we’d like.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#102

Post by jgrahamc »

Tossrock wrote: post #99
eastdakota wrote: post #13

Earlier quoted context omitted.

We thought we did. And tried both public and private lines of communication — without reply. Still waiting.

I love the inclusion of the "help4u@verizon.com" address in the email attempting to get their attention on a Severity 0 event. It was worth a shot!

We tweeted at @Verizon and @VerizonSupport also.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#103

Post by londons_explore »

Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours.

I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a watered down version with all specifics removed is published.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#104

Post by inopinatus »

eropple wrote: post #4

Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2…

Back in that oft-forgotten age I was privileged to know, work, and chill with the brave volunteers (and, subsequently, paid RIR staff) holding down the buttons. It’s worth mentioning that even back then there dwelt in the west a large ugly troll whose name was AS701 (AS701 was not alone, either, having two hideous siblings, AS702 and AS703, that lived in other climes). Everyone tiptoed around the beast, because when angered it would flap its routes and there would be a great wailing and severe packet loss. The brave volunteers tried many times to tame the awful creature and I’m very sorry to see that it is still fucking everyone’s announcements even today.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#105

Post by BuildTheRobots »

azernik wrote: post #98
woliveirajr wrote: post #17

> One of our network engineers made contact with DQE Communications quickly and after a little delay they were able to put us in contact with someone who could fix the problem. DQE worked with us on the phone to stop advertising these “optimized” routes to Allegheny Technologies Inc. We're grateful for their help. Once this was done, the Internet stabilized, and things went back to normal. It's funny how we have to s…

Gotta have a channel that's out-of-band with the internet to fix problems with the internet.

I'd love to know how out-of-band telephone networks (landline and mobile) actually are any more. A surprising amount goes via SIP on public IPs.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#106

Post by wil421 »

>It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place.

I see you haven’t had much experience with large Telcos. They are all like this.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#107

Post by icedchai »

yingw787 wrote: post #23
ziddoap wrote: post #6

Thank you for the summary. And, a sincere thank you for not mincing words when it comes to something as important as this. > However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare. > IRR filtering would not have increased Verizon's costs or limited their service in any way…

Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

The happens all the time to lesser degrees and is a fact of life on the Internet. No action will be taken.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#108

Post by ubercow13 »

psim1 wrote: post #57
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

I think it makes them look pretty fine to very good

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#109

Post by w8vY7ER »

tssva wrote: post #86

AS701 was the UUNET/Worldcom AS for the US and eventually the US/Canada network. Verizon bought Worldcom in 2006 and they became Verizon Business. From the late 90s to early 2000s I worked for UUNET/Worldcom as an engineer in the network planning and design group. I worked in the international group but among other things we were responsible for the build out of AS701 into Canada, the exchange sites where AS701 conne…

As they should, this is day-one BGP configuration stuff. Nothing advanced, nothing especially technical or time consuming. Speaks to a gross degree of professional negligence on the part of AS701. Really disappointing to see this degree of ignorance.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#110

Post by txcwpalpha »

psim1 wrote: post #57
eastdakota wrote: post #40

Earlier quoted context omitted.

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional.

What strikes me the most is that this whole "event" would have hardly even registered on anyone's radar (it affected less than 10% of their traffic during early hours of the morning. I saw one single news article about it, buried on The Verge, but other than that nothing), except for the fact that Cloudflare's CTO was on HN this morning fanning the flames of the one thread about it. It's like they dug their own hole drawing attention to the "Cloudflare outtage" headline, and now they're overcompensating by going to drastic measures to blame someone else.

And now they keep harping on the fact that Verizon still hasn't responded? Sure, part of that is probably the fact that Verizon is a giant corporation that doesn't want to bother with this stuff, but the other part is that this "event" was hardly even big enough of a deal to register on VZ's PR team's radar, no matter how much CF whines about it.

This blog post (and the accompanying HN comments from Cloudflare execs) just scream "immature company" to me. There's a reason that Cloudflare is the one making this blog post and devoting CEO time to it while the established behemoth is just going about their business as usual.

Post reply on HN