Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare silently deleted my DNS records

txti.es

Post reply on HN
Print view View original

321–330 of 433 posts

Re: Cloudflare silently deleted my DNS records

#321

Post by Aeolun »

DangerousPie wrote: post #307
jiggawatts wrote: post #279

Earlier quoted context omitted.

Please explain something to me. For me and my of my customers, having your "entire cloud deleted" is like... the #1 nightmare scenario. So why does this capability/function even exist for active accounts at CloudFlare? It sounds like the OP fell victim to what is essentially a regular process. Or to put it another way: No amount of explanation or assurance is ever going to make me feel comfortable with my doctor havi…

It doesn't sound like it actually deleted anything other than the DNS entries. It's not like any actual data got lost.

I dunno, but DNS entries are actual data.

I certainly don’t know what all the DNs entries on my 15 domains are, so if they disappear it’s not going to be easy to get it all back.

Re: Cloudflare silently deleted my DNS records

#322

Post by p1necone »

p1necone wrote: post #243
jgrahamc wrote: post #83

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve. Your solution basically boils down to "companies are failing at escalated support issues well, so they should escalate support issues well."

I posted this under the wrong parent comment, moved but cant delete.

Re: Cloudflare silently deleted my DNS records

#323

Post by bauerd »

Mojah wrote: post #189

Occasions are rare where I get to say "hey, I built a thing that might help here!" - so forgive me as I take this opportunity with both hands. Whether this was a bug or a rare protective mechanisme, there will be times when your DNS provider makes a mistake and removes records. You mentioned in your post your DNS isn't hard to reproduce, but how certain are you that _all_ records are restored? How long do you have to…

>but how certain are you that _all_ records are restored?

A solution to this is to keep DNS under version control with eg Terraform and deployed by CI. master is then authoritative

Re: Cloudflare silently deleted my DNS records

#324

Post by tialaramex »

gregd wrote: post #259
andrewstuart wrote: post #125

Earlier quoted context omitted.

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoi…

A properly setup helpdesk negates the need for an Ombudsman. If a tech company cannot get a helpdesk escalation path correct, what makes you think they'll get an Ombudsman scenario correct?

Ombudsman services are able to resolve conflicts where the customer and business don't agree. They're appropriate for regulated businesses to draw a line in the sand as to what is a legitimate complaint by independently settling cases where the customer and business did not agree. If the Ombudsman hears 100 unresolved complaints and upholds only one then probably the ordinary customer services are working as intended. If they uphold ninety nine something is gravely wrong with your customer services and the regulator needs to step in.

This isn't so dissimilar from the method used in pathology to deliver consistency in results across multiple labs and assay methods. You identify some boundary cases. This is definitely normal, this is definitely cancer but this is borderline, repeat a few times. You replicate and get all the labs to mark your samples. Then you can identify labs (or assay techniques) that aren't reliably putting things in the right categories and demand they improve or stop.

Re: Cloudflare silently deleted my DNS records

#325

Post by Aeolun »

jgrahamc wrote: post #166
gist wrote: post #134

Earlier quoted context omitted.

> BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people. I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a compan…

I disagree with you. It matters enormously that people like me are available. Sure there are time wasters who’ll send me email. But I don’t care. Dealing with a small number of real customers doesn’t take a lot of my time and matters. Everyone optimizes for the worst case. They think “if I give out my email I’ll get tons of useless email”. I can assure you I get 10x the crap sent to me on LinkedIn than via direct ema…

If I ever have someones direct email I don’t think I would use it in any situation that I wouldn’t appreciate getting one myself.

Re: Cloudflare silently deleted my DNS records

#326

Post by ex3ndr »

TylerE wrote: post #277
nickjj wrote: post #233

Earlier quoted context omitted.

> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved. What if the cost was put onto the business instead of the consumer and the business just hired support people who are all Ombudsmans by default? Instead of focusing on copy / pasting boilerplate scripts and answering as many tickets as possible, they should focus on the problem the custom…

Because that isn't a useful allocation of resources. Support that powerful are going to basically be devs. With dev salary expectations.

Yeah, even russian social network (200m users) have a bunch of very smart people supporting users from password recovery to psychological help. No way big startup can't afford something like this. You just have to do so.

Re: Cloudflare silently deleted my DNS records

#327

Post by IggleSniggle »

nickjj wrote: post #313
TylerE wrote: post #277

Earlier quoted context omitted.

Because that isn't a useful allocation of resources. Support that powerful are going to basically be devs. With dev salary expectations.

I think it's useful and it's very much worth it to give support a salary where they are enthusiastic to be employed. If someone uses your product or service and has an issue, the first contact they have with your company is through support. A crappy support experience could easily be the difference between having a life time customer valued at thousands of dollars, or your customer feeling frustrated and going to ano…

I think this happens because it's very very difficult for a manager's manager to know if the frontline employee is spending their time solving real problems or slacking off with no actual regard for the customer.

"Metrics" are an unhappy middleground for everyone.

Re: Cloudflare silently deleted my DNS records

#328

Post by gjs278 »

diegoperini wrote: post #281
stedaniels wrote: post #255

Earlier quoted context omitted.

jgrahamc is the CTO of Cloudflare. His only supervisor is eastdakota/ Matthew Prince, the CEO of Cloudflare.

Then I'd like to know the CEO's opinion then.

he doesn’t care at all. you are asking really bizarre questions.

Re: Cloudflare silently deleted my DNS records

#329

Post by joecool1029 »

andrewstuart wrote: post #125
jgrahamc wrote: post #83

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoi…

> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.

At least in the case of telecom companies they all seem to have executive response teams when the usual channels fail. Send an email to any of their Exec team members (CEO, president, CTO, etc) and it nearly always gets assigned to a special team that solves the problem. I can confirm this out of personal experience with at least AT&T, Comcast, and T-Mobile.

Re: Cloudflare silently deleted my DNS records

#330

Post by phyzome »

nickjj wrote: post #233
andrewstuart wrote: post #125

Earlier quoted context omitted.

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoi…

> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved. What if the cost was put onto the business instead of the consumer and the business just hired support people who are all Ombudsmans by default? Instead of focusing on copy / pasting boilerplate scripts and answering as many tickets as possible, they should focus on the problem the custom…

NearlyFreeSpeech.net does something kind of like that. It costs money to raise any support ticket (you have to have a special support subscription) but you get refunded your "support points" if it turned out to be a problem on their end.
Post reply on HN