Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

blog.cloudflare.com

Post reply on HN
Print view View original

261–270 of 291 posts

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#261

Post by bogomipz »

Cpoll wrote: post #256
bogomipz wrote: post #126

From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…

> regular people with kids and families and feelings. This is just an appeal to emotion. No-one is even calling out any individual people. With a company of this scale and responsibility, individuals shouldn't even come into the discussion, and there should be multiple levels of redundancy. Verizon, collectively, is being shamed. Verizon should be compared to a power plant, not a SaaS provider or some 3-person dev sh…

>"This is just an appeal to emotion."

Not at all, its an appeal to civility. The statement that Cloudflare made with "there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place' is the appeal to emotion here.

>"No-one is even calling out any individual people.'

No, that is a very clear attempt to call out a specific group of people who work in the network engineering department.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#262

Post by bogomipz »

jgrahamc wrote: post #228
bogomipz wrote: post #126

From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…

No one has forgotten Cloudbleed. It's something we talk about internally and every single day I look at a report that shows me status of software running around the world so that I never, ever again let a piece of software running on our edge crash and leak information.

The point wasn't whether or not Cloudbleed was forgotten but rather acting with some civility towards other when these mistakes do happen. Your suggestion that "there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place" is absurd. Misconfigurations and mistakes are a fact of life, they happen to everyone. To suggest that AS 701 which is old enough to have a 3 digit ASN, somehow doesn't use any ingress prefix filtering as a matter of course is disingenuous at best. The cause is quite likely that this was a misconfiguration on a single interface on a single router. I think you know this though.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#263

Post by jgrahamc »

bogomipz wrote: post #262
jgrahamc wrote: post #228

Earlier quoted context omitted.

No one has forgotten Cloudbleed. It's something we talk about internally and every single day I look at a report that shows me status of software running around the world so that I never, ever again let a piece of software running on our edge crash and leak information.

The point wasn't whether or not Cloudbleed was forgotten but rather acting with some civility towards other when these mistakes do happen. Your suggestion that "there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place" is absurd. Misconfigurations and mistakes are a fact of life, they happen to everyone. To suggest that AS 701 which is old enough to have a 3 digit ASN, s…

Yeah, because everyone was so civil towards me and Cloudflare when Cloudbleed happened.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#264

Post by bogomipz »

jgrahamc wrote: post #263
bogomipz wrote: post #262

Earlier quoted context omitted.

The point wasn't whether or not Cloudbleed was forgotten but rather acting with some civility towards other when these mistakes do happen. Your suggestion that "there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place" is absurd. Misconfigurations and mistakes are a fact of life, they happen to everyone. To suggest that AS 701 which is old enough to have a 3 digit ASN, s…

Yeah, because everyone was so civil towards me and Cloudflare when Cloudbleed happened.

Wow. So you've decided to propagate only the negative behavior from that incident? "Someone did it to me so I'm going to do it to someone else" - is that the thinking? There were also no shortage of people who showed support and understanding for your folks during that time(me included.)

What's the old adage - "be the change you want to see in the world." Seems like a real missed leadership opportunity.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#265

Post by [deleted] »

btown wrote: post #179
anonbutler wrote: post #152

Earlier quoted context omitted.

Tom is based in London. So he had a good night's sleep and was well rested.

We don’t know that he wasn’t up late fixing another bug that we thankfully never saw, and that his life hasn’t been like a season of 24 this past day.

[deleted]

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#266

Post by nikisweeting »

taildrop wrote: post #41
driverdan wrote: post #30

Earlier quoted context omitted.

It doesn't need government intervention. It needs other companies to hold them accountable.

Absolutely. If you are a Verizon service provider customer, please call them and register your feelings on this matter. Make sure you let them know in no uncertain terms that you are considering switching providers based on their lack of following best practices.

I did this yesterday, I called and expressed strong concern over Verizon's incident response and BGP security in general. I said I would not longer even want to be a FiOS customer, let alone business customer if I know that Verizon doesn't do basic prefix filtering on their BGP peers/customers.

I was careful not to berate/blame the T1 support people who have no clue what BGP is or even that an incident happened, but I tried to express the severity of the issue well enough that they would escalate a serious complaint to the network infra team.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#267

Post by nikisweeting »

icedchai wrote: post #21

I remember the early days of the Internet, when I could log in to an ISP router running BGP, with a blazing fast T1 to an early tier 1 Internet provider. We could literally announce any route we wanted, no filtering. We used to regularly black hole spammers, then turn them back on an hour or two later.

Please write about your life / these times! Or link me to your blog/writing if you already have. <3

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#268

Post by nikisweeting »

InTheArena wrote: post #173

I don’’t think Cloudflare is going to get any business from Verizon anytime soon. This may be as a professional of a “Hey Verizon, you don’t know what the fsck you are doing” as I’ve seen.

They're both big enough that they probably can't live without each other, which makes for an interesting relationship because they can probably swear at each other all day long and nothing will come of it.

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#269

Post by nikisweeting »

jgrahamc wrote: post #263
bogomipz wrote: post #262

Earlier quoted context omitted.

The point wasn't whether or not Cloudbleed was forgotten but rather acting with some civility towards other when these mistakes do happen. Your suggestion that "there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place" is absurd. Misconfigurations and mistakes are a fact of life, they happen to everyone. To suggest that AS 701 which is old enough to have a 3 digit ASN, s…

Yeah, because everyone was so civil towards me and Cloudflare when Cloudbleed happened.

(You handled both admirably well, I would just stop responding to this person.)

Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline

#270

Post by Dylan16807 »

txcwpalpha wrote: post #241
Dylan16807 wrote: post #240

Earlier quoted context omitted.

> And yet blaming a specific team is exactly what they did. In this specific case, just blaming "Verizon", it was not personal. (There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay.) Knowing it's the NOC team, as an amorphous blob of nameless people, is not getting too personal. Just because something can be traced to a team doesn't mean that shaming…

>In this specific case, just blaming "Verizon", it was not personal. That isn't what they did. They specifically called out teams, which according to what you just said, is too personal. https://twitter.com/eastdakota/status/1143182575680143361 > The teams at @verizon and @noction should be incredibly embarrassed at their failings this morning ... It’s networking malpractice that the NOC at @verizon has still not rep…

> according to what you just said, is too personal

That is not what I said!

I said it can be, and then I clarified with: There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay.

I see the tweet. I call this case not personal. He's pointing the blame at large groups inside someone else's opaque company.

If you're pointing at a blob of 100+ people (like you said, support is also being blamed) then you're not making it personal.

> Was there anything in this situation that was gained by Prince calling these people out in these tweets?

People know what company to blame (a good thing), but nobody outside that company even knows how many teams, let alone specifics about the people on those teams (an acceptable thing). Overall positive.

> Would it not have been just as effective at calling out Verizon (while being less unprofessional and less personally malicious) if those tweets had been less vitriolic?

Being less vitriolic would not make it more or less personally targeted.

I'm not sure if the vitriol helped exactly but I think Verizon did enough to deserve it that there's no need to berate Cloudflare for the vitriol itself.

> Why does CF have a license to be petty but VZ apparently does not?

Presuming I even agree with your definition of pettiness, the problem is not the pettiness itself, but the actions they take or don't take.

It's not terrible for VZ to be petty as long as they still fix their broken equipment.

Post reply on HN