Nothing is ever new in IT. A similar incident happened 22 years ago. Buggy and/or misconfigured gear disaggregates and reannounces routes and creates a blackhole.
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
131–140 of 291 posts
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#132I appreciate how sympathetic Cloudflare is to the root-cause party because they answered the phone and undid what they shouldn’t have done. (If my understanding is correct, they shouldn’t have told Verizon about the better routing, while Verizon should have known better)
I read this as Allegheny's fault, actually. DQE published to Allegheny (DQE's customer), who in turn re-published to Verizon (Allegheny's other provider). While most of the 'prevention' section talks about what Verizon didn't do, it doesn't seem to mention that Allegheny should not have re-published the DQE-published routes up to Verizon.
It's startling that Verizon doesn't appear to have any leak mitigations in place, but I feel like Allegheny is getting a pass here because they are small, or something.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#133Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours . I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a water…
Blog posts are their speciality
(deleted)
Yeah, that wasn’t contributing. Everyone has bad days.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#134Here's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.
I needed to burn some vacation time before EoY when it expires, so I started taking Mondays off once a month. Today was such a day. But I was supposed to be on-call, so I traded a day with a teammate.
I went off-call at 8:30am EST. Then, while the internet burned down, I slept in and played video games.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#135Earlier quoted context omitted.
Amen. We need a support group. "Hi, I'm Teejmya, and I was on call last night"
Yes, apologies and thank you! If you email me (matthewatcloudflaredotcom) your shirt size, preference for men's or women's cut, and your postal address with the subject line: "Verizon BGP Leak On Call Support Group" I'll send you a Cloudflare tshirt. Least we can do.
This is so awesome.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#136Earlier quoted context omitted.
I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional. What strikes me the most is that this whole "event" woul…
The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that: 1. They implement basic precautions to prevent dumb things from going wrong. 2. They're available 24/7, to immediately respond to and remediate whatever does go wrong. 3. Both of the above…
I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I don't care about? Who's fault it was. There are multiple buckets of companies here:
1. Cloud providers that were effected enough to apparently devote not insignificant CEO and CTO time to it (Cloudflare)
2. Cloud providers that were affected but seemingly not enough for it to even register as anything more than a blip on their status tracker (Google, AWS, etc)
3. Cloud providers that weren't effected
As a potential customer thinking about buying services from one of these companies, which one do you think I am doing to do with? It certainly won't be CF. And if I am already engaged with CF, I want to know what CF is going to do to mitigate this situation in the future, and no, pointing fingers like a child and saying "it wasn't our fault!" doesn't count.
Cloudflare can't really control Verizon's actions that lead to this situation, but they can control how they respond to it and mitigate it. They had an opportunity to stand up as a leader and improve the internet (which is literally their company motto). As you pointed out, the internet working correctly is a matter of companies working together as good actors, and getting these companies to work together via good, strong relationships is a part of that.
Did Cloudflare do that? Nah. Instead, they made a petty blog post and their CEO is on Twitter telling Verizon they should be ashamed. I don't know exactly what his goal there was, but I assume it has something to do with hoping they'll be better in the future (if that's not his goal, then it really is just petty finger pointing). And if Cloudflare's CEO's method of getting people to improve their work is to publicly shame them, I really feel bad for anyone who works under him.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#137Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#138Earlier quoted context omitted.
Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.
Hmm. My issue with that is a majority consensus could decide that, for instance, .gov domains should no longer be relegated to the American government. Same issue as with a crypto 51% takeover.
> Hmm. My issue with that is a majority consensus could decide that, for instance, .gov domains should no longer be relegated to the American government. Same issue as with a crypto 51% takeover.
I'm sure other countries would like to use .gov.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#139Earlier quoted context omitted.
Amen. We need a support group. "Hi, I'm Teejmya, and I was on call last night"
Yes, apologies and thank you! If you email me (matthewatcloudflaredotcom) your shirt size, preference for men's or women's cut, and your postal address with the subject line: "Verizon BGP Leak On Call Support Group" I'll send you a Cloudflare tshirt. Least we can do.
I think somebody should make some "I Fixed the Internet after Verizon Broke It. 20190624" T-shirts.
Re: Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
#140Earlier quoted context omitted.
Gotta have a channel that's out-of-band with the internet to fix problems with the internet.
Nowadays with voip “the phone” isn’t as out of band as we’d like.
Is it time to put an HF ham radio rig in each major provider’s office? I shudder thinking of a major outage where even phone communication can’t take place.
I’m only half joking.
Edit: and maybe we just give Verizon a toy walkie talkie