141–150 of 291 posts
My favorite outage when I worked for a voip company was when one of our tech support people told a new customer that she needed to ‘add our ip address to your router’, meaning add it to the firewall whitelist, but she repeated that verbatim to the telco tech who misunderstood and then escalated her way up the chain at a major telco until some engineer with the wrong rights said ‘fuck it’ and updated bgp to route all…
Earlier quoted context omitted.
So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.
Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.
All is not lost though. You can always opt out and run your own DNS or use hostfiles. Then you can have the internet you want and everyone the can have the internet they want.
Here's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.
Edit/Disclaimer: Yes, this is a joke. I woke up to several serious alarms just as the problem was starting. Luckily, I thought to check Cloudflare’s status page from my phone around the second or third time PagerDuty called me. I saw a preliminary notice from them indicating that they were observing networking issues. At that point, I decided I’d rather watch the world burn from my bed than my computer, so I “scheduled” maintenance for a few hours and went back to bed. Our whole infrastructure had split by that point, but there was nothing I could do about it.
Earlier quoted context omitted.
So we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.
Freedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.
Earlier quoted context omitted.
The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.
I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional. What strikes me the most is that this whole "event" woul…
Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours . I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a water…
Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortun…
Earlier quoted context omitted.
Blog posts are their speciality
(deleted) Yeah, that wasn’t contributing. Everyone has bad days.
> Chief Architect, Information Security, Akamai Technologies. I do not speak for my employer.
Probably best to disclose this more directly in comments on topics related to competitors.
It's clear that Verizon broke the Internet this morning through incompetent BGP management, and they could do it again. Who holds them accountable?
Who holds them accountable? Anyone working in their NOCs with thoughts of working elsewhere? Having a predetermination of "stupid and/or lazy" because of your workplace can't help employment prospects.
> Anyone working in their NOCs with thoughts of working elsewhere? Having a predetermination of "stupid and/or lazy" because of your workplace can't help employment prospects.
Some people select workplaces based on the notion that they are lazy.
I appreciate how sympathetic Cloudflare is to the root-cause party because they answered the phone and undid what they shouldn’t have done. (If my understanding is correct, they shouldn’t have told Verizon about the better routing, while Verizon should have known better)
I read this as Allegheny's fault, actually. DQE published to Allegheny (DQE's customer), who in turn re-published to Verizon (Allegheny's other provider). While most of the 'prevention' section talks about what Verizon didn't do, it doesn't seem to mention that Allegheny should not have re-published the DQE-published routes up to Verizon. It's startling that Verizon doesn't appear to have any leak mitigations in plac…
(I’m from Pittsburgh, my grandfather and a bunch of my relatives worked for this company for decades. I’ve been kinda giggling about this intersection of my past and my present all day. I don’t work in the parts of Cloudflare that deal with this kind of thing; I’m glad my co-workers were on top of it.)