Cloudflare outage on June 21, 2022 • Hacker News Board

Re: Cloudflare outage on June 21, 2022

#221

Post by [deleted] » Wed, Jun 22, 2022, 5:00 AM UTC

bandwidth wrote: post #220

kebab-case wrote: post #209

I lead the platform team of a fairly young startup in the D2C commerce space in the APAC region. This outage happened during peak traffic hours which made me and the team look like amateurs in the company. Cloudflare is great, and I would never move away from it. But from a business continuity standpoint, is there a fallback approach that we should be prepared for during such cases? One crude approach we were discuss…

If you're okay with load balancing DNS queries across multiple providers you could do 2x Cloudflare primary NS, 2x GCP for example where each provider is in sync with each other. If not, a manual swap at the registrar level would be good enough. I should also mention this approach sort of breaks with Cloudflare's proxied records which dynamically assign anycast IPs for records placed on their CDN. So if using this ap…

[deleted]

Re: Cloudflare outage on June 21, 2022

#222

Post by viraptor » Wed, Jun 22, 2022, 5:20 AM UTC

Uehreka wrote: post #112

viraptor wrote: post #19

Earlier quoted context omitted.

I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture…

Ehhhh… I think it’s good (for us) that they do this, but I don’t think it’s a free ad (contrary to popular belief, not all news is good news, and this is bad news) and any sort of conversion rate on recruitment is probably vanishingly small (which would normally be fine, but incidents like these may turn off some actual customers, which is where actual revenue comes from). I think their calculation (to the extent you…

> but incidents like these may turn off some actual customers

Incidents - yes. But why would a post-mortem turn someone off? The incident happened regardless. Do you think anyone would be more likely turned off by reading how they solved it / plan to prevent it on the future than by silence?

Re: Cloudflare outage on June 21, 2022

#223

Post by weird-eye-issue » Wed, Jun 22, 2022, 6:06 AM UTC

algo_trader wrote: post #91

weird-eye-issue wrote: post #75

One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money. Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage. So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudf…

Could you possibly, kindly, mention which tools you use to track/buy/calculate conversions/revenue? Many thanks (Or DM the puppet email in my profile)

Our revenue is reported to us by our ad network and Amazon Associates. Basic events in Google Analytics let's us see various types of conversions

Re: Cloudflare outage on June 21, 2022

#224

Post by AtNightWeCode » Wed, Jun 22, 2022, 6:43 AM UTC

Would be nice to have some automation that one could use for keeping track of health status of cloud services. Status API, webhook solution, something. Maybe even a standard for it. Or a service that monitors all major cloud services.

We did get alarms. Our things partially worked though so CF was not the first thing to check.

Re: Cloudflare outage on June 21, 2022

#225

Post by michaelmior » Wed, Jun 22, 2022, 7:14 AM UTC

scottlamb wrote: post #140

michaelmior wrote: post #53

Earlier quoted context omitted.

This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.

I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.

You're probably right, but if they're going to mention it as part of the problem, I would want to see it as part of the solution. However, I agree that they certainly shouldn't name names.

Re: Cloudflare outage on June 21, 2022

#226

Post by can16358p » Wed, Jun 22, 2022, 7:54 AM UTC

stnmtn wrote: post #149

dylan604 wrote: post #131

Earlier quoted context omitted.

"Is it Apple Maps bad?" --Gavin Belson, Silicon Valley This one line will forever cement exactly how bad Apple Maps' release was. Thanks Mike Judge!

I agree, but lately (as in the past month) I've been finding myself using apple maps more and more than google. When on a complicated highway interchange, the 3d view that Apple Maps gives for which exit to take is a life-saver

Recently I used Apple Maps much more than Google Maps.

In addition to trying to de-Googlify my life, there was also an occurance where Google Maps literally tried to kill me: at an intersection that connects into a highway it guided me to drive straight into the opposite direction to a highway, straight onto the coming cars at 140km/h. I've quit Google Maps right there and never used it again.

Re: Cloudflare outage on June 21, 2022

#227

Post by vnkr » Wed, Jun 22, 2022, 11:52 AM UTC

daniel-grigg wrote: post #199

llama052 wrote: post #189

We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing.. It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet. I wish more companies would take note of the transparency and sense o…

Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.

Cloudflare's status page is one of the very few services of Cloudflare that are hosted elsewhere precisely for the reason it needs to be available if something isn't working.

Re: Cloudflare outage on June 21, 2022

#228

Post by devonkim » Wed, Jun 22, 2022, 12:57 PM UTC

Most of the criticisms seem to be around BGP and network management. What I’m seeing here that also is important is that the change was applied to a DC where the route change didn’t trigger the defect. In essence, this is also due to a very classic problem of a test dataset giving a false sense of security due to variation from other configurations. For this reason my team prefers to rollout changes to production using a test region that most customers don’t use yet will have some visible impact if there’s any error in our presumptions so far such as hard-coding regions and relying upon services not present or as capable across all regions. This practice has caught a number of rather serious errors for us that while customer impacting was nowhere near as bad as if we had rolled out simply randomly like many teams do essentially. This is even more important the more difficult it is to perform rollbacks of changes or for rollbacks to take effect such as DNS and CDN caching changes.

Re: Cloudflare outage on June 21, 2022

#229

Post by WorldMaker » Wed, Jun 22, 2022, 5:52 PM UTC

techsupporter wrote: post #160

forrestthewoods wrote: post #144

Earlier quoted context omitted.

Some tools are more fragile and error prone than others.

Except that this wasn't an example of BGP being prone to error or fragile. This was, as the blog post specifically calls out, human error. They put two BGP announcement rules after the "deny everything not previously allowed" rule. It's the same as if someone did this to a set of ACLs on a firewall. The main difference between BGP and all other tools is that if you mess up BGP, you've done a very visible thing becaus…

That does seem like bad UX/"DevX" that that configuration of rules is "valid" syntactically and there weren't better equivalents of "linters"/"compilers" flagging that before it ever got sent out as an announcement. UX issues are a "proneness" to error/fragility. It sounds like there is room to build a "higher level language" (like a "Typescript : Javascript :: ? : BGP") for BGP announcements that is less prone to "accidentally bad programs". Not that I have immediate suggestions, just that my gut reaction from skimming these sorts of outage reports is that if it was a "language" I was writing in I can hear that I'd want a lot more (type) safety nets.

Re: Cloudflare outage on June 21, 2022

#230

Post by wondernine » Thu, Jun 23, 2022, 12:06 AM UTC

Part of the blog says :

"In this time, we’ve converted 19 of our data centers to this architecture, internally called Multi-Colo PoP (MCP): Amsterdam, Atlanta, Ashburn, Chicago, Frankfurt, London, Los Angeles, Madrid, Manchester, Miami, Milan, Mumbai, Newark, Osaka, São Paulo, San Jose, Singapore, Sydney, Tokyo."

Is the term MCP synonymous with "tier 1 PoPs" (mentioned elsewhere in other cloudflare blogs from time to time) or are the two terms referring to different things?