Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

221–230 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#221

Post by eecc »

txcwpalpha wrote: post #136
SpicyLemonZest wrote: post #129

Earlier quoted context omitted.

The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that: 1. They implement basic precautions to prevent dumb things from going wrong. 2. They're available 24/7, to immediately respond to and remediate whatever does go wrong. 3. Both of the above…

I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…

So your definition of successful and mature business is a management team that doesn’t give 2f?

Wow

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#222

Post by shakna »

bogomipz wrote: post #202
shakna wrote: post #193

Earlier quoted context omitted.

Cloudflare reached out multiple times in multiple ways to Verizon, to attempt to resolve the situation. More than eight hours on, after utilising everything from what they were told was a Tier 1 support line to Twitter, they have nothing. Even if we're kind to Verizon about the network failure, which was a global issue, they haven't done anything or said anything to suggest that Cloudflare should be treating them kin…

Have you ever worked for a Tier 1 ISP during a big outage? There is not enough personnel bandwidth in a NOC for everyone to get an individual response. >"Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons" Oh the Cloudflare marketing machine. Largest by "utilized"? What does that even mean? Cloudflare is not a Tier 1, a Tier 2, or a major eyeball net…

> The fact that you have taken this so personally is kind of embarrassing.

What? I have said nothing personal.

> What this blog post, the opportunistic marketing ploy and finger pointing have shown is a complete lack of maturity on your part.

Ah. You seem to be confused. I am not affiliated with Cloudflare, and have not worked with Cloudflare at any point in time.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#223

Post by txcwpalpha »

Dylan16807 wrote: post #218
txcwpalpha wrote: post #211

Earlier quoted context omitted.

Yes, and? Cloudflare themselves are the ones pushing their own company as "leaders" in this field, and being a "leader" does not mean "pointing fingers and trying to avoid blame whenever something bad happens". If they fancy themselves leaders regarding BGP, as said on their website, then they need to actually act like leaders. And as I've said multiple times now, Cloudflare was in a great position here to stand them…

Companies respond just fine to public scrutiny, caused by them being rightfully and loudly blamed. The way you lead people isn't the same as the way you lead companies. Verizon was acting so badly that it's clear the pure friendly approach was doing absolutely nothing. And I'm sure Cloudflare is willing to give very real and pleasant engineering help if desired. If Verizon doesn't want to talk to Cloudflare, that's f…

>rightfully and loudly blamed

There is an enormous difference between assigning fault in a good faith attempt to find a root cause/solution, and casting unnecessary, unprofessional insults such as "Verizon's team should be ashamed of themselves". One is productive, and the other is just being a dick.

>The way you lead people isn't the same as the way you lead companies.

Yes, it certainly is. A company is an organization of people, after all. You don't get to eschew professionalism and start throwing around insults just because a group of people has decided to attach an additional label over their heads.

And just to put an even finer point on it, Matthew Prince's tweets about the issue were not targeted at Verizon "the company". He specifically attacked Verizon's NOC and its team members. Despite everything, this isn't a faceless, soulless corporation that's having insults hurled at them. He specifically went after a specific group of people and publicly shamed them. And then he has the gall to shame them even more for not immediately chomping at the bit to help someone who just aggressively insulted them.

Ask yourself: if Matthew Prince had sent a tweet berating team members from his own company, telling them they should be ashamed of themselves, and spent the rest of the day commenting on the internet insulting their competence, would you still be saying he is a good leader? Or even a good CEO? Of course not. It's Leadership 101 that insulting your team members isn't a good leadership style. And that doesn't change just because Prince isn't the one signing the Verizon team's paychecks.

> This is not a problem that requires active cooperation.

This is clearly not the opinion of those at Cloudflare that are loudly kicking their feet and whining that Verizon didn't devote enough resources to actively cooperate with Cloudflare's troubleshooting today.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#224

Post by threezero »

bogomipz wrote: post #170
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

I’d say that Verizon’s lack of cooperation was devoid of any respect or professionalism.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#225

Post by threezero »

bogomipz wrote: post #202
shakna wrote: post #193

Earlier quoted context omitted.

Cloudflare reached out multiple times in multiple ways to Verizon, to attempt to resolve the situation. More than eight hours on, after utilising everything from what they were told was a Tier 1 support line to Twitter, they have nothing. Even if we're kind to Verizon about the network failure, which was a global issue, they haven't done anything or said anything to suggest that Cloudflare should be treating them kin…

Have you ever worked for a Tier 1 ISP during a big outage? There is not enough personnel bandwidth in a NOC for everyone to get an individual response. >"Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons" Oh the Cloudflare marketing machine. Largest by "utilized"? What does that even mean? Cloudflare is not a Tier 1, a Tier 2, or a major eyeball net…

This is a pretty rude response to someone who doesn’t even work for cloudflare.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#226

Post by jgrahamc »

jawns wrote: post #146
londons_explore wrote: post #103

Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours . I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a water…

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

The sequence of events went a bit like this:

Team in London started working the problem and called in reinforcements from elsewhere;

Upper management (me and one other person) got involved as it was serious/not resolved fast;

I spoke with the network team in London who seemed to have a good handle on the problem and how they were working to resolve but we decided to wake a couple of other smart folks up to make sure we had all the best people on it;

Problem got resolved by the diligence of an engineer in London getting through to and talking with DQE;

Some people went back to bed;

Tom worked on writing our internal incident report so that details were captured fast and people had visibility. He then volunteered to be point on writing the public blog (1415 UTC);

Folks in California woke up and got involved with the blog. Ton of people contributed to it from around the world with Tom fielding all the changes and ideas;

Very senior people at Cloudflare (including legal) signed off and we posted (1958 UTC).

No one had an axe to grind with Verizon. We were working a complex problem affecting a good chunk of our traffic and customers. Everyone was calm and collected and thoughtful throughout.

Shout out to the Support team who handled an additional 1,000 support requests during the incident!

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#227

Post by oasisbob »

bogomipz wrote: post #170
jawns wrote: post #146

Earlier quoted context omitted.

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues…

> You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers.

What element of the blog post are you referring to? NANOG often speaks in jargon and obtuse-professional speak, but with large routing leaks there are always strong opinions expressed. It's been this way going back well over a decade.

Another counterexample: go search the NANOG archives for opinions on AWS, EC2, and SES. You won't find much reciprocal respect - you'll find a bunch of unabashed criticism on how AWS operates, and how that affects the internet.

This is a clash of cultures. Cloudflare knows their customers expect a fast, accurate, transparent explanation. NANOG participants are used to an environment where their dirty laundry isn't aired in public to the point where they get calls from reporters asking about it.

Cloudflare is walking a tight line where they're trying to accurately explain to a lay audience what happened to their customers. They can't assume their audience knows what AS 701 is, or BCP 38, or the DFZ, or the prior harm that BGP optimizers have been known to cause.

A "professional" NANOG thread would touch on all of that, it just wouldn't be pieced together under a single byline for a mass audience.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#228

Post by jgrahamc »

bogomipz wrote: post #126

From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…

No one has forgotten Cloudbleed. It's something we talk about internally and every single day I look at a report that shows me status of software running around the world so that I never, ever again let a piece of software running on our edge crash and leak information.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#229

Post by jgrahamc »

txcwpalpha wrote: post #110
psim1 wrote: post #57

Earlier quoted context omitted.

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional. What strikes me the most is that this whole "event" woul…

You should go back and read what I wrote: https://news.ycombinator.com/item?id=20262316

I didn't fan flames. There was already a link to our status page on the front page of HN. While the event was happening I gave short updates by editing a comment here.

Also, your "affected less than 10% of their traffic during early hours of the morning" is incredibly parochial and seems to ignore the fact that people use the Internet world over.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#230

Post by txcwpalpha »

jgrahamc wrote: post #229
txcwpalpha wrote: post #110

Earlier quoted context omitted.

I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional. What strikes me the most is that this whole "event" woul…

You should go back and read what I wrote: https://news.ycombinator.com/item?id=20262316 I didn't fan flames. There was already a link to our status page on the front page of HN. While the event was happening I gave short updates by editing a comment here. Also, your "affected less than 10% of their traffic during early hours of the morning" is incredibly parochial and seems to ignore the fact that people use the Inte…

>While the event was happening I gave short updates by editing a comment here.

It is disingenuous to only state you edited "a comment" there. You posted 10 comments in that thread, with at least another 10 edits. Of the top 5 comments, three of them are yours. On HN, each time you make a comment and people upvote your comment, it contributes to ranking the post higher on HN's front page. I fully understand that you were probably just trying to be communicative, but unintentionally or not, you did "fan the flames" by drawing additional attention to the issue.

Post reply on HN