Cloudflare silently deleted my DNS records • Hacker News Board

Re: Cloudflare silently deleted my DNS records

#261

Post by p0sixlang » Mon, Feb 24, 2020, 9:40 PM UTC

andrewstuart wrote: post #125

jgrahamc wrote: post #83

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoi…

Some companies, this doesn't work at all. EG: Postmates. Emailed pretty much every one of their executive/management team about a literally brand damaging issue, and received zero bounces, but also zero replies. Some companies have a policy to ignore unsolicited emails, no matter how serious the issue, as to not fuel the idea that doing so will get results in the future.

Re: Cloudflare silently deleted my DNS records

#262

Post by AviationAtom » Mon, Feb 24, 2020, 9:42 PM UTC

iudqnolq wrote: post #114

jgrahamc wrote: post #109

Earlier quoted context omitted.

I’ve worked for Cloudflare since it was 24 people. I care a hell of a lot about our customers. I know eastdakota does also.

I can tell. Cloudflare has been a delight to use up to now and I'm grateful I'm getting this kind of support when I've only used your loss-leader services.

You mention loss-leader services, but the model is no doubt setup to funnel you towards profitable services.

I switched a personal domain to them, but it won't be long before I get my employer to move their resources over to them, assuming I see this all play out well.

Re: Cloudflare silently deleted my DNS records

#263

Post by AviationAtom » Mon, Feb 24, 2020, 9:44 PM UTC

jgrahamc wrote: post #103

iudqnolq wrote: post #94

Earlier quoted context omitted.

OP here. You can post updates with any relevant information. Probably goes without saying, but if the issue has to do with my billing or address please don't post specific details without asking me first. I will link to this comment from TFA for verification. (Edit: added to the bottom. If you need more verification you have my email.) Edit2: I see that the domain is back in my account and listed as "Pending Nameserv…

Thanks. Appreciate it. I’ll let the team look into it and communicate with you first.

John, you guys seem pretty awesome about posting root cause analysis and being transparent, can we expect to see at least a comment (if not a blog post) summarizing what took place here?

Re: Cloudflare silently deleted my DNS records

#264

Post by mercora » Mon, Feb 24, 2020, 9:45 PM UTC

iudqnolq wrote: post #172

mercora wrote: post #130

Earlier quoted context omitted.

i think the parent poster meant changes done by or for a registrar like shown in whois. Zone changes wont show up there.

Cloudflare is my register and manages my DNS. I would expect anything them to log any significant changes to either in their audit log.

i think that it would be rather unusual to update timestamps in whois (which i guessed the parent poster was referring to) based on updates to in-zone data. A transfer would be an example of something that would change information in whois and thus update the timestamp noted there for the latest update. it is sometime possible to infer the date of the latest change of in-zone data because the serial of the zone is often constructed by using a date and a counter. But that is actually just convention and not reliable. Its also unlikely the parent poster was referring to this.

Re: Cloudflare silently deleted my DNS records

#265

Post by RcouF1uZ4gsC » Mon, Feb 24, 2020, 9:54 PM UTC

andrewstuart wrote: post #125

jgrahamc wrote: post #83

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoi…

> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.

HN actually acts somewhat like a crowd-sourced ombudsman. People who have an issue write a description and post it to HN. If enough people find it compelling, it makes the front page. Once it makes the front page, someone in authority at the involved tech company will see it, and try their best to resolve it.

Re: Cloudflare silently deleted my DNS records

#266

Post by tolstoshev » Mon, Feb 24, 2020, 9:55 PM UTC

andrewstuart wrote: post #125

jgrahamc wrote: post #83

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoi…

That often ends up being the defacto job of a online community manager, if they have one.

Re: Cloudflare silently deleted my DNS records

#267

Post by iudqnolq » Mon, Feb 24, 2020, 9:55 PM UTC

petre wrote: post #191

iudqnolq wrote: post #122

Earlier quoted context omitted.

I would try to set up a completely open source monitoring setup just for fun, but once I'm paying for SNS I personally would rather just pay epsilon more and buy/rent the whole system. I get that may just be personal taste. I absolutely don't trust myself to run my own highly-reliable mailserver to send status alerts.

Just send them locally and pull them with IMAP onto your phone.

I didn't even know that was possible. Thanks for teaching me something new. It's always nice to learn I need less SAAS magic than I thought.

Re: Cloudflare silently deleted my DNS records

#268

Post by HBKXNCUO » Mon, Feb 24, 2020, 9:55 PM UTC

craftinator wrote: post #229

djsumdog wrote: post #17

I really had a strong dislike for Cloudflare after they banned certain customers for political reasons[1]. The CEO mentioned how maybe it wasn't the right thing to do .. and then they did it again. There aren't really any self-hosted solutions for DDoS protection like Cloudflare since it requires things happening in the network layer. Implementing a solution would require access to monitor and reshape the local netwo…

If you run a restaurant, you can refuse to do business with anyone you choose. If that was not the case, you would effectively be a slave; unable to choose actions for yourself and your business. Cloudflare refused to do business with people and content; that is their prerogative.

>If you run a restaurant, you can refuse to do business with anyone you choose.

You will suffer legal consequences if it's determined that you refused to do business with them on the basis of certain characteristics protected by law. Property rights are not absolute. Exceptions to them can be made if people think there is good reason to do so.

The situation we are in now, where technology companies have found themselves with wide power to control the public political debate occurring among regular people merely as a consequence of successfully running some particular types of business, is not one we've really seen before. There are some very persuasive arguments for limiting their property rights, similar to how they were limited e.g. ~50 years ago by the civil rights act.

Re: Cloudflare silently deleted my DNS records

#269

Post by b112 » Mon, Feb 24, 2020, 9:56 PM UTC

ignoramous wrote: post #162

briandear wrote: post #140

Earlier quoted context omitted.

> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. All companies should have that!

In Amazon's case, Bezos' email is just that. For AWS, Andy Jassy's (CEO) or Charlie Bell's (SVP) might do just as fine. Source: Someone who emails them from time to time to get impasses resolved.

Just had to chime in here, but fifteen+ years ago, Bezo's email was ombudsman like, with him or a top level person reading it.

Now, it's just another level of very poor, very scripted support.

I'm fairly sure Amazon has taken the (perhaps wise, in a business sense) approach of not caring if a small percentage of users leave, due to support issues.

The cost of keeping customers with certain support issues, greatly outweighing supporting them.

This is why you have to hunt madly around Amazon's webpage to find contact info, why all forms of help point away from contacting a person, including their chat being bots now, until you move outside of their scripts.

Re: Cloudflare silently deleted my DNS records

#270

Post by iudqnolq » Mon, Feb 24, 2020, 9:56 PM UTC

isclever wrote: post #222

iudqnolq wrote: post #75

Earlier quoted context omitted.

> Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare. I believe it was both. If I have a registrar outage I'm hosed. If I don't have a registrar outage and do have a DNS outage I can recover with a little work. But in the only case I…

A domain registered at a provider (but not DNS) can be down with no impact to your domain, so long as the domain is still in the TLD root servers, everything will keep going.

Thank you for teaching me something new. I didn't know that got cached.