Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

231–240 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#231

Post by jgrahamc »

txcwpalpha wrote: post #230
jgrahamc wrote: post #229

Earlier quoted context omitted.

You should go back and read what I wrote: https://news.ycombinator.com/item?id=20262316 I didn't fan flames. There was already a link to our status page on the front page of HN. While the event was happening I gave short updates by editing a comment here. Also, your "affected less than 10% of their traffic during early hours of the morning" is incredibly parochial and seems to ignore the fact that people use the Inte…

>While the event was happening I gave short updates by editing a comment here. It is disingenuous to only state you edited "a comment" there. You posted 10 comments in that thread, with at least another 10 edits. Of the top 5 comments, three of them are yours. On HN, each time you make a comment and people upvote your comment, it contributes to ranking the post higher on HN's front page. I fully understand that you w…

So, you'd propose I sit back and leave a story with incomplete information on HN's front page and say nothing?

True that I posted other comments but they are short and don't say much. The real action was the main top comment.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#232

Post by txcwpalpha »

jgrahamc wrote: post #231
txcwpalpha wrote: post #230

Earlier quoted context omitted.

>While the event was happening I gave short updates by editing a comment here. It is disingenuous to only state you edited "a comment" there. You posted 10 comments in that thread, with at least another 10 edits. Of the top 5 comments, three of them are yours. On HN, each time you make a comment and people upvote your comment, it contributes to ranking the post higher on HN's front page. I fully understand that you w…

So, you'd propose I sit back and leave a story with incomplete information on HN's front page and say nothing? True that I posted other comments but they are short and don't say much. The real action was the main top comment.

Absolutely not! I appreciate your communication during the incident, and I definitely don't mean to discourage any participation in threads or reduce communication. I'm just pointing out that it did, if unintentionally, draw more attention to the issue.

What I don't appreciate is your company's unprofessional response re: Verizon after the issue ended, but that's been discussed elsewhere.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#233

Post by Dylan16807 »

txcwpalpha wrote: post #223
Dylan16807 wrote: post #218

Earlier quoted context omitted.

Companies respond just fine to public scrutiny, caused by them being rightfully and loudly blamed. The way you lead people isn't the same as the way you lead companies. Verizon was acting so badly that it's clear the pure friendly approach was doing absolutely nothing. And I'm sure Cloudflare is willing to give very real and pleasant engineering help if desired. If Verizon doesn't want to talk to Cloudflare, that's f…

>rightfully and loudly blamed There is an enormous difference between assigning fault in a good faith attempt to find a root cause/solution, and casting unnecessary, unprofessional insults such as "Verizon's team should be ashamed of themselves". One is productive, and the other is just being a dick. >The way you lead people isn't the same as the way you lead companies. Yes, it certainly is. A company is an organizat…

> A company is an organization of people

Blaming a specific team can get too personal. Blaming an entire company is more about the decision-making structure, and is close to as impersonal as you can get. It's really not the same as blaming a person.

> This is clearly not the opinion of those at Cloudflare that are loudly kicking their feet and whining that Verizon didn't devote enough resources to actively cooperate with Cloudflare's troubleshooting today.

They didn't notice, acknowledge, or fix the problem. That's different from a lack of resources devoted to active cooperation. Heck, two messages of "on it" and "it's fixed" would be a pleasant level of "active cooperation", and that takes only a minute or two.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#234

Post by txcwpalpha »

Dylan16807 wrote: post #233
txcwpalpha wrote: post #223

Earlier quoted context omitted.

>rightfully and loudly blamed There is an enormous difference between assigning fault in a good faith attempt to find a root cause/solution, and casting unnecessary, unprofessional insults such as "Verizon's team should be ashamed of themselves". One is productive, and the other is just being a dick. >The way you lead people isn't the same as the way you lead companies. Yes, it certainly is. A company is an organizat…

> A company is an organization of people Blaming a specific team can get too personal. Blaming an entire company is more about the decision-making structure, and is close to as impersonal as you can get. It's really not the same as blaming a person. > This is clearly not the opinion of those at Cloudflare that are loudly kicking their feet and whining that Verizon didn't devote enough resources to actively cooperate…

> Blaming a specific team can get too personal.

And yet blaming a specific team is exactly what they did.

>They didn't notice, acknowledge, or fix the problem. That's different from a lack of resources devoted to active cooperation. Heck, two messages of "on it" and "it's fixed" would be a pleasant level of "active cooperation", and that takes only a minute or two.

Sure, I'm not defending Verizon's inaction. My point is that regardless of the level of the cooperation, some cooperation is clearly still required. And now because of Cloudflare's hostility towards Verizon after this incident, I wouldn't be surprised if Verizon is much less inclined to participate in any cooperation. That not only seems counterproductive to Cloudflare's goal, it's also bad for all of us that use the internet.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#235

Post by lima »

jgrahamc wrote: post #226
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

The sequence of events went a bit like this: Team in London started working the problem and called in reinforcements from elsewhere; Upper management (me and one other person) got involved as it was serious/not resolved fast; I spoke with the network team in London who seemed to have a good handle on the problem and how they were working to resolve but we decided to wake a couple of other smart folks up to make sure…

Thank you! CloudFlare's response is appropriate.

The incident itself and lack of response (for HOURS) from Verizon's side is absolutely unacceptable. It's 2019, filtering ALL of your customer's routes according to - at least - the IRR (including the legacy ones connected to the old router in the closet) and having a responsive 24/7 NOC contact in PeeringDB are a matter of course.

Proper carriers like NTT go above and beyond simple IRR filtering nowadays with things like peerlock (http://instituut.net/~job/peerlock_manual.pdf).

AT&T uses RPKI and was completely unaffected: https://twitter.com/Jerome_UZ/status/1143276134907305984

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#236

Post by irjustin »

jgrahamc wrote: post #226
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

The sequence of events went a bit like this: Team in London started working the problem and called in reinforcements from elsewhere; Upper management (me and one other person) got involved as it was serious/not resolved fast; I spoke with the network team in London who seemed to have a good handle on the problem and how they were working to resolve but we decided to wake a couple of other smart folks up to make sure…

Awesome. Thanks for this timeline and for the team being absolutely amazing.

I love the shaming of Verizon without the sugar coat. Divisive for sure, but a welcomed one.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#237

Post by lima »

eastdakota wrote: post #124
kevinreedy wrote: post #62

> The RPKI framework that we implemented and deployed globally last year is designed to prevent this type of leak. It enables filtering on origin network and prefix size. The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is. Does RPKI prevent Cloudflare from announcing additional /22 routes during an inci…

We could break our prefixes into smaller routes, but 1) the Internet's routers have limited memory; 2) we have a lot of routes; and 3) we want to be good Internet citizens. If every network announced all their routes as /24s — the smallest route generally accepted over the public Internet — the routing table would be a giant mess and would overwhelm many routers' ability to store them. That said, after today we are t…

Kudos for not deaggregating routes into /24s like many other major ISPs do nowadays.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#238

Post by lima »

bogomipz wrote: post #126

From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…

Mistakes happen and CloudFlare's response to the memory leak was excellent.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#239

Post by lima »

rayvd wrote: post #195
bogomipz wrote: post #170

Earlier quoted context omitted.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

https://mailman.nanog.org/pipermail/nanog/2019-June/101614.h...

Weird response by the Verizon employee.

> You guys have repeatedly accused them of being dumb without even speaking to anyone yet from the sounds of it.

Not for lack of trying...

> Should they have been easier to reach once an issue was detected? Probably. They’re certainly not the first vendor to have a slow response time though. Seems like when an APAC carrier takes 18 hours to get back to us, we write it off as the cost of doing business.

It wasn't a slow response, it was no response. And either is unacceptable for a tier 1 carrier.

> But this industry is one big ass glass house. What’s that thing about stones again?

And other carriers are actively working to change that - including, in particular, CloudFlare.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#240

Post by Dylan16807 »

txcwpalpha wrote: post #234
Dylan16807 wrote: post #233

Earlier quoted context omitted.

> A company is an organization of people Blaming a specific team can get too personal. Blaming an entire company is more about the decision-making structure, and is close to as impersonal as you can get. It's really not the same as blaming a person. > This is clearly not the opinion of those at Cloudflare that are loudly kicking their feet and whining that Verizon didn't devote enough resources to actively cooperate…

> Blaming a specific team can get too personal. And yet blaming a specific team is exactly what they did. >They didn't notice, acknowledge, or fix the problem. That's different from a lack of resources devoted to active cooperation. Heck, two messages of "on it" and "it's fixed" would be a pleasant level of "active cooperation", and that takes only a minute or two. Sure, I'm not defending Verizon's inaction. My point…

> And yet blaming a specific team is exactly what they did.

In this specific case, just blaming "Verizon", it was not personal. (There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay.)

Knowing it's the NOC team, as an amorphous blob of nameless people, is not getting too personal.

Just because something can be traced to a team doesn't mean that shaming the company is the same as shaming specific people from that team.

Going down that road would declare everything as personal, and that's really not how things work.

> I wouldn't be surprised if Verizon is much less inclined to participate in any cooperation.

The public pressure should be stronger than any pettiness, and if it's not then the solution is to let even more people know it was Verizon's fault.

Post reply on HN