Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

191–200 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#191

Post by bogomipz »

FussyZeus wrote: post #187
bogomipz wrote: post #180

Earlier quoted context omitted.

Do you really think that "throwing shade" is what the internet needs? Is "throwing shade" an admirable quality in someone who is supposed to be demonstrating leadership? Anyone who has worked as a network engineer for a major ISP knows the internet is quite brittle. During my entire time in that profession I can't remember a time when attempting to shame people was used to resolve a routing issue or to improve relati…

Shame is one of the most effective tools in influencing human behavior, and from the sounds of this post and the other coverage on the incident, Verizon has earned far more ire than is directed at them in this blog post. A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course;…

How about leaking session tokens and other sensitive data for millions of people during "Cloudbleed"? Were you advocating public shaming for Cloudflare then? Was that also "sheer laziness" and did they earn "more than a few" of your "go fuck yourself's"?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#192

Post by kortex »

heywire wrote: post #140
jedberg wrote: post #101

Earlier quoted context omitted.

Nowadays with voip “the phone” isn’t as out of band as we’d like.

Is it time to put an HF ham radio rig in each major provider’s office? I shudder thinking of a major outage where even phone communication can’t take place. I’m only half joking. Edit: and maybe we just give Verizon a toy walkie talkie

You joke, but cascade failure ain't no laughing matter, either. Emergency plans aren't for fair weather, they are for sh!tstorms. I don't want to say "we are too reliant on the internet" because it's cliche and connectivity is just part of growing the modern world. But we sure as heck need several layers of backup plans in case things go sideways.

I, for one, hope there is a secret society of HAMs lurking as mild-mannered employees at every telco and ISP, ready to wire things back together when they short out.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#193

Post by shakna »

bogomipz wrote: post #126

From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…

Cloudflare reached out multiple times in multiple ways to Verizon, to attempt to resolve the situation.

More than eight hours on, after utilising everything from what they were told was a Tier 1 support line to Twitter, they have nothing.

Even if we're kind to Verizon about the network failure, which was a global issue, they haven't done anything or said anything to suggest that Cloudflare should be treating them kindly in any way.

Not even a "we're aware, we're handling it".

Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons.

Verizon have shown a complete lack of leadership.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#194

Post by kyuudou »

inopinatus wrote: post #104
eropple wrote: post #4

Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2…

Back in that oft-forgotten age I was privileged to know, work, and chill with the brave volunteers (and, subsequently, paid RIR staff) holding down the buttons. It’s worth mentioning that even back then there dwelt in the west a large ugly troll whose name was AS701 (AS701 was not alone, either, having two hideous siblings, AS702 and AS703, that lived in other climes). Everyone tiptoed around the beast, because when…

uunet , alternet and ? now it's verizon eh

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#195

Post by rayvd »

bogomipz wrote: post #170
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

https://mailman.nanog.org/pipermail/nanog/2019-June/101614.h...

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#196

Post by Gaelan »

inopinatus wrote: post #104
eropple wrote: post #4

Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2…

Back in that oft-forgotten age I was privileged to know, work, and chill with the brave volunteers (and, subsequently, paid RIR staff) holding down the buttons. It’s worth mentioning that even back then there dwelt in the west a large ugly troll whose name was AS701 (AS701 was not alone, either, having two hideous siblings, AS702 and AS703, that lived in other climes). Everyone tiptoed around the beast, because when…

For the lazy:

AS701 Verizon Business/UUnet

AS702 Verizon Business/UUnet Europe

AS703 Verizon Business/UUnet ASPAC

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#197

Post by ethbro »

bin0 wrote: post #128
ethbro wrote: post #120

Earlier quoted context omitted.

Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.

Hmm. My issue with that is a majority consensus could decide that, for instance, .gov domains should no longer be relegated to the American government. Same issue as with a crypto 51% takeover.

Web of trust is not consensus.

Fragmenting and an inability to formulate a standard answer when communicating are serious issues with using it, but a takeover is not.

If Chinese users would rather trust some Chinese government entity to resolve URLs, then that's what they'd get. At the same time as other users might get something else.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#198

Post by [deleted] »

bogomipz wrote: post #191
FussyZeus wrote: post #187

Earlier quoted context omitted.

Shame is one of the most effective tools in influencing human behavior, and from the sounds of this post and the other coverage on the incident, Verizon has earned far more ire than is directed at them in this blog post. A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course;…

How about leaking session tokens and other sensitive data for millions of people during "Cloudbleed"? Were you advocating public shaming for Cloudflare then? Was that also "sheer laziness" and did they earn "more than a few" of your "go fuck yourself's"?

[deleted]

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#199

Post by [deleted] »

txcwpalpha wrote: post #189
SpicyLemonZest wrote: post #154

Earlier quoted context omitted.

To be frank, your post makes it clear that you don't know the context. CF simply cannot do anything on their own to mitigate the problem where Verizon constructs bad BGP routes to Cloudflare IPs and then advertises those routes to third parties. The only mitigation possible is to contact whoever's advertising the bad routes and get them to stop.

Have you read Cloudflare's multiple blog posts regarding BGP? Did you read the tweets from their directors talking about how other customers were unaffected by the event today because of the mitigations put in place? Did you even do the simplest Google about BGP protocols and the plans in place to prevent this from happening in the future? If you're going to try to impose yourself as the gatekeeper of "knowing the co…

[deleted]

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#200

Post by Cpoll »

bogomipz wrote: post #177
MrStonedOne wrote: post #147

Earlier quoted context omitted.

It was gross negligence that leads to this. Nothing more.

Do you work for Verizon's NOC or Network Engineering department then? You have inside knowledge that it was negligence? Because I provided a specific scenario where it would not be "gross negligence."

> Do you work for Verizon's NOC or Network Engineering department then

If that's what it takes to be able to decide, then I guess we can safely declare that it wasn't negligence, because no-one that fits that description will ever publicly admit to it.

Post reply on HN