Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

241–250 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#241

Post by txcwpalpha »

Dylan16807 wrote: post #240
txcwpalpha wrote: post #234

Earlier quoted context omitted.

> Blaming a specific team can get too personal. And yet blaming a specific team is exactly what they did. >They didn't notice, acknowledge, or fix the problem. That's different from a lack of resources devoted to active cooperation. Heck, two messages of "on it" and "it's fixed" would be a pleasant level of "active cooperation", and that takes only a minute or two. Sure, I'm not defending Verizon's inaction. My point…

> And yet blaming a specific team is exactly what they did. In this specific case, just blaming "Verizon", it was not personal. (There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay.) Knowing it's the NOC team, as an amorphous blob of nameless people, is not getting too personal. Just because something can be traced to a team doesn't mean that shaming…

>In this specific case, just blaming "Verizon", it was not personal.

That isn't what they did. They specifically called out teams, which according to what you just said, is too personal.

https://twitter.com/eastdakota/status/1143182575680143361

> The teams at @verizon and @noction should be incredibly embarrassed at their failings this morning ... It’s networking malpractice that the NOC at @verizon has still not replied to messages

Not only does he specifically call out the NOC, he also calls out teams. It is very obvious which "the teams" he is referring to, and "the NOC" is indeed a specific team. In other comments he also calls out Verizon's support team.

This wasn't the case of "tracing it back to a team". CF's CEO specifically addressed them and told them to be ashamed of themselves. That's personal, and it's also being a dick to boot. Was there anything in this situation that was gained by Prince calling these people out in these tweets? Would it not have been just as effective at calling out Verizon (while being less unprofessional and less personally malicious) if those tweets had been less vitriolic?

> The public pressure should be stronger than any pettiness, and if it's not then the solution is to let even more people know it was Verizon's fault.

So the solution to pettiness is more pettiness? Why does CF have a license to be petty but VZ apparently does not?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#242

Post by hn20180220 »

IntelMiner wrote: post #210
dreamcompiler wrote: post #139

Earlier quoted context omitted.

I think somebody should make some "I Fixed the Internet after Verizon Broke It. 20190624" T-shirts.

"Verizon broke the internet and all I got was this lousy T-shirt"

I would need a bunch of them

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#243

Post by hn20180220 »

eastdakota wrote: post #124
kevinreedy wrote: post #62

> The RPKI framework that we implemented and deployed globally last year is designed to prevent this type of leak. It enables filtering on origin network and prefix size. The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is. Does RPKI prevent Cloudflare from announcing additional /22 routes during an inci…

We could break our prefixes into smaller routes, but 1) the Internet's routers have limited memory; 2) we have a lot of routes; and 3) we want to be good Internet citizens. If every network announced all their routes as /24s — the smallest route generally accepted over the public Internet — the routing table would be a giant mess and would overwhelm many routers' ability to store them. That said, after today we are t…

Kudos for a CEO that understands in and outs of Internet routing, making me want to join CF's neteng team

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#245

Post by LoSboccacc »

btown wrote: post #179
anonbutler wrote: post #152

Earlier quoted context omitted.

Tom is based in London. So he had a good night's sleep and was well rested.

We don’t know that he wasn’t up late fixing another bug that we thankfully never saw, and that his life hasn’t been like a season of 24 this past day.

we don't know either if he's an alien coming from a planet that has a shorter day and his circadian cycle is being disrupted being on earth, but Occam would suggest no.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#246

Post by cwilkes »

coldreactor wrote: post #119
teejmya wrote: post #77

Earlier quoted context omitted.

Amen. We need a support group. "Hi, I'm Teejmya, and I was on call last night"

"I'm Coldreactor and I was traumatized by the massive number of calls last night"

“Or I would have been if routing was working correctly”

There’s a silver lining to everything.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#247

Post by bin0 »

vageli wrote: post #138
bin0 wrote: post #128

Earlier quoted context omitted.

Hmm. My issue with that is a majority consensus could decide that, for instance, .gov domains should no longer be relegated to the American government. Same issue as with a crypto 51% takeover.

> Hmm. My issue with that is a majority consensus could decide that, for instance, .gov domains should no longer be relegated to the American government. Same issue as with a crypto 51% takeover. I'm sure other countries would like to use .gov.

They are entitled to their opinion; I saw a lot of this on the .amazon thread yesterday. However, right now, it's used by America only. There are some perks to inventing the internet.

My point is I don't like a system in which the majority can decide they don't like you having something and take it. For instance, what if people decided they didn't like facebook, so decided to seize its domain?

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#248

Post by notacoward »

KirinDave wrote: post #182
owenmarshall wrote: post #178

Earlier quoted context omitted.

> behave however they like and we all simply have to deal with it So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??

Not really. You don't have to be a massive player to screw things up with BGP.

You don't need to be a massive player to initiate the screwup, but you kind of need a massive player like Verizon to amplify it for you. That's why the onus on them should be greater.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#249

Post by bogomipz »

Varriount wrote: post #220
bogomipz wrote: post #191

Earlier quoted context omitted.

How about leaking session tokens and other sensitive data for millions of people during "Cloudbleed"? Were you advocating public shaming for Cloudflare then? Was that also "sheer laziness" and did they earn "more than a few" of your "go fuck yourself's"?

I fail to see how Cloudbleed and this event are the same. Cloudbleed was caused by a Cloudflare bug, true, but it wasn't caused by outright laziness (which this incident clearly was). Furthermore, unlike Verizon's distinct lack of communication regarding this incident, Cloudflare has generally been very good about reporting and communicating with the community.

They are not same nor was I implying they were the same. The point being that oversight and genuine mistakes happen. However Cloudflare wants to characterize it as "sloppiness and laziness" when it's someone else. And even here you are "parroting" them here when neither you or they actually know the details do you? Clearly Verizon has prefix filtering in place in other places or this would be a regular occurrence for them and it is not. Cloudflare is pretty far down the list in importance during an outage that affected many companies - other Tier 1s, 2s etc. Just because Cloudflare didn't get a response does not mean they weren't communicating. I know two network engineers who were in contact with Verizon yesterday during the outage. But again you seem intent to just "pile on" after reading a one-sided and self-serving Cloudflare blog post.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#250

Post by collinmanderson »

Varriount wrote: post #220
bogomipz wrote: post #191

Earlier quoted context omitted.

How about leaking session tokens and other sensitive data for millions of people during "Cloudbleed"? Were you advocating public shaming for Cloudflare then? Was that also "sheer laziness" and did they earn "more than a few" of your "go fuck yourself's"?

I fail to see how Cloudbleed and this event are the same. Cloudbleed was caused by a Cloudflare bug, true, but it wasn't caused by outright laziness (which this incident clearly was). Furthermore, unlike Verizon's distinct lack of communication regarding this incident, Cloudflare has generally been very good about reporting and communicating with the community.

> outright laziness (which this incident clearly was)

I don't think it was clearly laziness. It could have been a configuration mistake.

Post reply on HN