Has Verizon responded in any capacity yet?
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
271–280 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#272Earlier quoted context omitted.
>In this specific case, just blaming "Verizon", it was not personal. That isn't what they did. They specifically called out teams, which according to what you just said, is too personal. https://twitter.com/eastdakota/status/1143182575680143361 > The teams at @verizon and @noction should be incredibly embarrassed at their failings this morning ... It’s networking malpractice that the NOC at @verizon has still not rep…
> according to what you just said, is too personal That is not what I said! I said it can be, and then I clarified with: There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay. I see the tweet. I call this case not personal. He's pointing the blame at large groups inside someone else's opaque company. If you're pointing at a blob of 100+ people (like you…
>If you're pointing at a blob of 100+ people (like you said, support is also being blamed) then you're not making it personal.
Ahh, I see. So it's okay that he was offensive and insulting, because he was offensive and insulting to many people? It wouldn't have been okay if he was offensive and insulting to only a handful of people, but because it was more than that, it's okay? Is this some weird perversion of "one death is a tragedy, 1000 deaths is a statistic"?
He isn't pointing the blame at a large group inside an "opaque" company. He's insulting people. The people at Verizon will know full well that he is talking to them. People that work with the Verizon NOC will know full well that those specific people are being insulted by this CEO. The fact that it was personally directed at multiple people doesn't make it any less personal, it just makes it personal to more people, no matter how much you move the goalposts.
> I'm not sure if the vitriol helped exactly but I think Verizon did enough to deserve it that there's no need to berate Cloudflare for the vitriol itself.
So it didn't help to berate Verizon, but it was still okay because they "deserved it"? And then you don't apply the same logic to Cloudflare themselves? There absolutely is a need to berate Cloudflare for their unnecessary use of vitriol, especially if you're telling me the bar for berating someone is as low as "it didn't help but that's okay".
It's clear at this point that you're moving goalposts and adjusting your own principles in some weird attempt to defend Cloudflare. Cloudflare did nothing positive here, and your attempt to justify their vitriol and maliciousness is telling.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#273Earlier quoted context omitted.
Yeah, because everyone was so civil towards me and Cloudflare when Cloudbleed happened.
(You handled both admirably well, I would just stop responding to this person.)
I'm sorry is holding a differing opinion and discussing that in a civil manner such an issue for you that you feel compelled to step in and offer prescriptive advice? Long live the filter bubble I guess huh?
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#274Earlier quoted context omitted.
The sequence of events went a bit like this: Team in London started working the problem and called in reinforcements from elsewhere; Upper management (me and one other person) got involved as it was serious/not resolved fast; I spoke with the network team in London who seemed to have a good handle on the problem and how they were working to resolve but we decided to wake a couple of other smart folks up to make sure…
Thank you! CloudFlare's response is appropriate. The incident itself and lack of response (for HOURS) from Verizon's side is absolutely unacceptable. It's 2019, filtering ALL of your customer's routes according to - at least - the IRR (including the legacy ones connected to the old router in the closet) and having a responsive 24/7 NOC contact in PeeringDB are a matter of course. Proper carriers like NTT go above and…
So basically we should avoid all Verizon Enterprise Product.
EdgeCast used to be my favourite CDN. Not sure how they are doing now.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#275Earlier quoted context omitted.
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smalle…
I do love that the CEO of Cloudflare is throwing technical shade at Verizon and others here and on Twitter for being useless.
This is the whole reason CF will never be seen as professional. Incidents like this, blog posts that've clearly not been anywhere close to a PR department, the whole CEO-blocking-sites-he-doesn't-like incident.. no wonder Verizon ignored their emails.
CF is great if you need "free" protection for a pet project, not really anything more.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#276Earlier quoted context omitted.
I read this as Allegheny's fault, actually. DQE published to Allegheny (DQE's customer), who in turn re-published to Verizon (Allegheny's other provider). While most of the 'prevention' section talks about what Verizon didn't do, it doesn't seem to mention that Allegheny should not have re-published the DQE-published routes up to Verizon. It's startling that Verizon doesn't appear to have any leak mitigations in plac…
Allegheny is a steel company. I don’t think most people expect them to have the same responsibility for internet health as Verizon, even though they are a $4B company. (I’m from Pittsburgh, my grandfather and a bunch of my relatives worked for this company for decades. I’ve been kinda giggling about this intersection of my past and my present all day. I don’t work in the parts of Cloudflare that deal with this kind o…
I had to do a double take because they seem to be selling themselves as a technology company nowadays.
I guess “steel” in your name isn’t good for share prices.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#277Earlier quoted context omitted.
> according to what you just said, is too personal That is not what I said! I said it can be, and then I clarified with: There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay. I see the tweet. I call this case not personal. He's pointing the blame at large groups inside someone else's opaque company. If you're pointing at a blob of 100+ people (like you…
>If you're pointing at a blob of 100+ people (like you said, support is also being blamed) then you're not making it personal. Ahh, I see. So it's okay that he was offensive and insulting, because he was offensive and insulting to many people? It wouldn't have been okay if he was offensive and insulting to only a handful of people, but because it was more than that, it's okay? Is this some weird perversion of "one de…
> Is this some weird perversion of "one death is a tragedy, 1000 deaths is a statistic"?
Nah. If you deliver 100 insults to 100 people, that's terrible. But if you deliver one insult to a vague blob of 100 people, that barely registers. The amount of insult directed at any specific person is tiny. That's why I'm not bothered by it.
> it just makes it personal to more people
No.
> no matter how much you move the goalposts
Really?
Someone disagrees with you so they must be moving goalposts?
Do better than that. I've been consistent on what I consider personal.
Also, I think you're too focused on vitriol. You can single people out and cause them harm while using the nicest and most polite language in the world. The way you target and your underlying meaning is far more important than your choice of words.
> So it didn't help to berate Verizon, but it was still okay because they "deserved it"? And then you don't apply the same logic to Cloudflare themselves? There absolutely is a need to berate Cloudflare for their unnecessary use of vitriol, especially if you're telling me the bar for berating someone is as low as "it didn't help but that's okay".
Let's put it this way. I regard "impersonal beration" as one tenth the crime of "being obviously and extremely negligent with equipment that can break the internet". And I'm willing to forgive vitriol when it's deserved and impersonal.
You don't forgive that, and want to say Cloudflare acted somewhat badly? Okay, sure.
You want to claim they are failing as a leader, overcompensating with drastic childish measures to blame someone else for something they could and should have mitigated themselves? I completely disagree.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#278I remember the early days of the Internet, when I could log in to an ISP router running BGP, with a blazing fast T1 to an early tier 1 Internet provider. We could literally announce any route we wanted, no filtering. We used to regularly black hole spammers, then turn them back on an hour or two later.
Please write about your life / these times! Or link me to your blog/writing if you already have. <3
It's not terribly interesting. No blog, sorry.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#279Earlier quoted context omitted.
Weird response by the Verizon employee. > You guys have repeatedly accused them of being dumb without even speaking to anyone yet from the sounds of it. Not for lack of trying... > Should they have been easier to reach once an issue was detected? Probably. They’re certainly not the first vendor to have a slow response time though. Seems like when an APAC carrier takes 18 hours to get back to us, we write it off as th…
>"It wasn't a slow response, it was no response. And either is unacceptable for a tier 1 carrier." Have you dealt with a Tier 1 carrier before? If you are not a peer i.e another Tier 1 you don't get immediate responses even in the best of times. >"It wasn't a slow response, it was no response. And either is unacceptable for a tier 1 carrier." No. I know people at Telia and NTT who were in contact with Verizon during…
CloudFlare is not a tier 1 carrier if you go by the strict definition of the term, just like Google isn't - but it's one of the largest content networks, reponsible for a significant percentage of internet traffic, with a global carrier-level backbone. Google and CloudFlare even tend to have better internal routing than most tier 1 providers.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#280Earlier quoted context omitted.
Thank you! CloudFlare's response is appropriate. The incident itself and lack of response (for HOURS) from Verizon's side is absolutely unacceptable. It's 2019, filtering ALL of your customer's routes according to - at least - the IRR (including the legacy ones connected to the old router in the closet) and having a responsive 24/7 NOC contact in PeeringDB are a matter of course. Proper carriers like NTT go above and…
So basically we should avoid all Verizon Enterprise Product. EdgeCast used to be my favourite CDN. Not sure how they are doing now.
Keep in mind that Verizon is a huge organization