One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.
Makes me wonder whether Cloudflare will start running quarterly drills with all their alerting peers.
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
161–170 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#162Earlier quoted context omitted.
The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that: 1. They implement basic precautions to prevent dumb things from going wrong. 2. They're available 24/7, to immediately respond to and remediate whatever does go wrong. 3. Both of the above…
I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…
>Did Cloudflare do that?
Yes?
>Cloudflare has decided that it's high-time we took a leadership role to finally secure BGP routing
etc.
https://blog.cloudflare.com/rpki/
>their CEO is on Twitter telling Verizon they should be ashamed
Yes, well
>I'll be the first to line up for a good publish lashing of US ISPs
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#163Earlier quoted context omitted.
The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that: 1. They implement basic precautions to prevent dumb things from going wrong. 2. They're available 24/7, to immediately respond to and remediate whatever does go wrong. 3. Both of the above…
I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's faul…
As far as mitigations, you're demanding they magically fix something they don't control. It's kind of infuriating, really. Stop what you're doing and reassess, please.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#164Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours . I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a water…
Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
There’s no excuse for the personal slander here - “his beauty rest interrupted” is a mean-spirited, inappropriate way to discuss Cloudflare’s post. Please don’t insult people unnecessarily here.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#165Earlier quoted context omitted.
Your profile mentions: > Chief Architect, Information Security, Akamai Technologies. I do not speak for my employer. Probably best to disclose this more directly in comments on topics related to competitors.
Why? Comments don't typically disclose "I am a corporate spambot" and yet such things exist, so you should probably read every comment as though it has some angle / vested interest. At least brians was nice enough to mention their bias in their profile. Most cases won't be so easy. Making people "opt in" to disclosing biases will just make it easier for the real bad actors to slip through.
brians seems like a good actor, which is why I assumed they might appreciate the feedback.
I agree there will always be bad actors on hn, which is too bad. I think the moderators try hard to combat it, which I am grateful for.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#166Earlier quoted context omitted.
We thought we did. And tried both public and private lines of communication — without reply. Still waiting.
Hats off to Verizon for treating everyone equally without discrimination.
'Can you hear me now?'
......
'Can you hear me now?'
......
'Can you hear me now?'
6 hours later, when the 9-5p NOC wakes up....
'Can you hear me now?'
Oh, you were talking. Whoops!
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#167Earlier quoted context omitted.
Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
There’s no excuse for the personal slander here - “his beauty rest interrupted” is a mean-spirited, inappropriate way to discuss Cloudflare’s post. Please don’t insult people unnecessarily here.
Likewise, don't insinuate people are ugly or don't deserve sleep.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#168Earlier quoted context omitted.
Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
There’s no excuse for the personal slander here - “his beauty rest interrupted” is a mean-spirited, inappropriate way to discuss Cloudflare’s post. Please don’t insult people unnecessarily here.
Uh, I think the idiom just means “got woken up early.” I don’t see an attack there.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#169Earlier quoted context omitted.
It doesn't need government intervention. It needs other companies to hold them accountable.
Absolutely. If you are a Verizon service provider customer, please call them and register your feelings on this matter. Make sure you let them know in no uncertain terms that you are considering switching providers based on their lack of following best practices.
And enjoy their fantastic Super Complaint Automation Tasker system, which takes your valuable feedback and "$1 > /dev/null"
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#170Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours . I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a water…
Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them"
Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues since the beginning of the commercial internet. It is very much a "community" with reciprocal respect and professionalism, things this blog post was devoid of.