Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

251–260 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#251

Post by bogomipz »

lima wrote: post #239
rayvd wrote: post #195

Earlier quoted context omitted.

https://mailman.nanog.org/pipermail/nanog/2019-June/101614.h...

Weird response by the Verizon employee. > You guys have repeatedly accused them of being dumb without even speaking to anyone yet from the sounds of it. Not for lack of trying... > Should they have been easier to reach once an issue was detected? Probably. They’re certainly not the first vendor to have a slow response time though. Seems like when an APAC carrier takes 18 hours to get back to us, we write it off as th…

>"It wasn't a slow response, it was no response. And either is unacceptable for a tier 1 carrier."

Have you dealt with a Tier 1 carrier before? If you are not a peer i.e another Tier 1 you don't get immediate responses even in the best of times.

>"It wasn't a slow response, it was no response. And either is unacceptable for a tier 1 carrier."

No. I know people at Telia and NTT who were in contact with Verizon during the incident. Just because Cloudflare wasn't in touch with them does not mean there was no response.

>'And other carriers are actively working to change that - including, in particular, CloudFlare."

Cloudflare is not a carrier. It's surprising you don't know that and yet are so ready to criticize.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#252

Post by ryanlol »

rayvd wrote: post #195
bogomipz wrote: post #170

Earlier quoted context omitted.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

https://mailman.nanog.org/pipermail/nanog/2019-June/101614.h...

Whats up with the Verizon employee comparing AS701 to APAC carriers? That’s a super harsh thing to publicly say about your employer.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#253

Post by bogomipz »

shakna wrote: post #222
bogomipz wrote: post #202

Earlier quoted context omitted.

Have you ever worked for a Tier 1 ISP during a big outage? There is not enough personnel bandwidth in a NOC for everyone to get an individual response. >"Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons" Oh the Cloudflare marketing machine. Largest by "utilized"? What does that even mean? Cloudflare is not a Tier 1, a Tier 2, or a major eyeball net…

> The fact that you have taken this so personally is kind of embarrassing. What? I have said nothing personal. > What this blog post, the opportunistic marketing ploy and finger pointing have shown is a complete lack of maturity on your part. Ah. You seem to be confused. I am not affiliated with Cloudflare, and have not worked with Cloudflare at any point in time.

."Ah. You seem to be confused. I am not affiliated with Cloudflare, and have not worked with Cloudflare at any point in time."

So you are just parroting statements made by Cloudflare in a marketing-laden blog post but doing so in a matter of fact way? I see.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#254

Post by carry_bit »

heywire wrote: post #140
jedberg wrote: post #101

Earlier quoted context omitted.

Nowadays with voip “the phone” isn’t as out of band as we’d like.

Is it time to put an HF ham radio rig in each major provider’s office? I shudder thinking of a major outage where even phone communication can’t take place. I’m only half joking. Edit: and maybe we just give Verizon a toy walkie talkie

You can't use the amateur bands for commercial purposes though.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#255

Post by bogomipz »

wtallis wrote: post #205
bogomipz wrote: post #177

Earlier quoted context omitted.

Do you work for Verizon's NOC or Network Engineering department then? You have inside knowledge that it was negligence? Because I provided a specific scenario where it would not be "gross negligence."

> Because I provided a specific scenario where it would not be "gross negligence." No, you didn't. You provided a vague conjecture for how the initial cause of the problem might not have been gross negligence, but offered no hypothesis for why Verizon isn't answering the Red Phone.

Yes I did. It's the part where I clearly state it's possible that a router that offline or rebooted came up with a stale or incorrect config. This actually happens occasionally. I've been on both ends of it. Clearly Verizon has inbound prefix filtering in place as this is not some common occurrence for AS 701.

Verizon was in contact with people yesterday. I have spoken to two people from two other carriers who were in touch with them. And you are just parroting the idea that because Cloudflare didn't get a response that Verizon wasn't responding to anyone period. And that's just not true. The fact that you think there's some red phone that just anyone can call the NOC and magically speak to someone during a major outage shows you have no practical experience with thes things you are commenting on and criticizing.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#256

Post by Cpoll »

bogomipz wrote: post #126

From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…

> regular people with kids and families and feelings.

This is just an appeal to emotion. No-one is even calling out any individual people. With a company of this scale and responsibility, individuals shouldn't even come into the discussion, and there should be multiple levels of redundancy. Verizon, collectively, is being shamed.

Verizon should be compared to a power plant, not a SaaS provider or some 3-person dev shop.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#257

Post by bogomipz »

collinmanderson wrote: post #250
Varriount wrote: post #220

Earlier quoted context omitted.

I fail to see how Cloudbleed and this event are the same. Cloudbleed was caused by a Cloudflare bug, true, but it wasn't caused by outright laziness (which this incident clearly was). Furthermore, unlike Verizon's distinct lack of communication regarding this incident, Cloudflare has generally been very good about reporting and communicating with the community.

> outright laziness (which this incident clearly was) I don't think it was clearly laziness. It could have been a configuration mistake.

Indeed and we saw that as a cause recently for a major Google outage. However that likely possibility of a bad config or edit doesn't fit the narrative Cloudflare is spinning here - that Verizon is simply dumb and lazy.

Clearly Verizon has inbound prefix filtering in place otherwise this would be a common occurrence for AS 701 and it is most certainly not. And it's quite surprising and sad to see how willing people are to just blindly parrot Cloudflare here and pile on. This of course was the desired outcome of the blog post.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#258

Post by KirinDave »

takeda wrote: post #213
KirinDave wrote: post #181

Earlier quoted context omitted.

Wow I've got some beachfront property in Nevada to sell you if you think the "web of trust" actually addressed any credible threat model.

Not a threat model, but gives me freedom to decide whom I trust.

That's precisely what it doesn't do. It's a transitive trust relationship.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#259

Post by michaelt »

bogomipz wrote: post #251
lima wrote: post #239

Earlier quoted context omitted.

Weird response by the Verizon employee. > You guys have repeatedly accused them of being dumb without even speaking to anyone yet from the sounds of it. Not for lack of trying... > Should they have been easier to reach once an issue was detected? Probably. They’re certainly not the first vendor to have a slow response time though. Seems like when an APAC carrier takes 18 hours to get back to us, we write it off as th…

>"It wasn't a slow response, it was no response. And either is unacceptable for a tier 1 carrier." Have you dealt with a Tier 1 carrier before? If you are not a peer i.e another Tier 1 you don't get immediate responses even in the best of times. >"It wasn't a slow response, it was no response. And either is unacceptable for a tier 1 carrier." No. I know people at Telia and NTT who were in contact with Verizon during…

I think what lima is saying is the Verizon employee basically says "Why didn't you call us for comment before publicly complaining that we never answer our phones?"

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#260

Post by icedchai »

unethical_ban wrote: post #7

I am surprised that CF is as aggressive toward Verizon in public as they are. Once you start breaking the Internet for stupid reasons, though, you probably deserve it. I know very little about BGP operations; I did not know that there was PKI and route validation like they described in the article.

Almost nobody rejects invalid routes. RPKI is basically a research project given that 85% of routes don't even have ROAs. See https://rpki-monitor.antd.nist.gov/
Post reply on HN