Live data from Hacker News

Cloudflare silently deleted my DNS records

txti.es

251–260 of 433 posts

Re: Cloudflare silently deleted my DNS records

#252
post #218

Earlier quoted context omitted.

(OP here). That looks really useful. If I was running a real service I would definitely look into it. Because this is just the personal website and email of a college student I don't think I could justify the expense when using something like Uptime Robot to monitor if a single record points to a web server would probably give me close to the same reliability.

Oh I absolutely agree! If you're a business, whether it's a SaaS or "just" a marketing website for your brick & mortar store, I think it's crucial to have back-ups. Most people think of backups as files, database dumps, previous versions, etc of their website. But the configuration data (in the form of DNS) isn't often considered. You're tech-savvy and can restore your DNS records because you know yoru servers' IP ad…

[deleted]

Re: Cloudflare silently deleted my DNS records

#253

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

First, you are awesome, really :) Second, a bunch of honest questions: Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the indu…

The CTO at Cloudflare has to ask his supervisor before trying to help someone over at hacker news? :)

Re: Cloudflare silently deleted my DNS records

#254
post #236

Earlier quoted context omitted.

$20 won't stop anyone with using it. If it is refundable, make it $300. That is high enough that only people and businesses with a showstopper situation will use it.

Make it $20 but not refundable. Or $300 and not refundable. This is a good way to filter the full level of importance. Most places that need a problem solved, now, dammit will be willing to pay. And, if you want to be nice, you can refund it. Or, if the person was a jerk, you can keep it.

If I have a rightful complaint and it costs me $300 to resolve I am done with that business.

Re: Cloudflare silently deleted my DNS records

#255

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

First, you are awesome, really :) Second, a bunch of honest questions: Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the indu…

jgrahamc is the CTO of Cloudflare. His only supervisor is eastdakota/ Matthew Prince, the CEO of Cloudflare.

Re: Cloudflare silently deleted my DNS records

#256

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

Context for the lazy, jgrahamc is the longtime CTO of Cloudflare.

Re: Cloudflare silently deleted my DNS records

#257
Stories like these scare the hell out of me. What do you do if one of the big internet corporation deletes some resource or account that is critical to your business? What happens when support isn’t responsive and you don’t have contacts in the company or your HN post doesn’t get visibility?

I get it - these are free services. You should factor that into every decision. But the risk is real even if you pay for an account. I’ve been slowly moving away from Gmail to a custom domain, but something like loosing DNS records and not being able to restore them quickly is even worse.

Back up everything that can be backed up, don’t rely on a single provider and always have a continuity plan!

Re: Cloudflare silently deleted my DNS records

#258

Earlier quoted context omitted.

You need to host across several nodes in different geographic locations and data centers to resist network splits. Then you need some way to slowly roll out upgrades to your monitoring platform over time.

I'm just talking about my personal infrastructure. If I host my crap in Vultr or Linode, I should be able to buy one cheap node on another provider just to run a simple status app: something with celary or sidekiq jobs to check my other stuff and intervals and generate a page with some red/yellow/green dots.

How do you know if the monitoring node goes down at the same time as the other servers?

Remember that Linode/Vultr/etc don't run their own datacenters, they share datacenters and sometimes downtime events can exist outside of datacenters.

Re: Cloudflare silently deleted my DNS records

#259

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoi…

A properly setup helpdesk negates the need for an Ombudsman. If a tech company cannot get a helpdesk escalation path correct, what makes you think they'll get an Ombudsman scenario correct?

Re: Cloudflare silently deleted my DNS records

#260
post #189

Occasions are rare where I get to say "hey, I built a thing that might help here!" - so forgive me as I take this opportunity with both hands. Whether this was a bug or a rare protective mechanisme, there will be times when your DNS provider makes a mistake and removes records. You mentioned in your post your DNS isn't hard to reproduce, but how certain are you that _all_ records are restored? How long do you have to…

The issue I see with this is that

1) You can't use it after the fact.

2) It's very specialized.

People are not going to set up dozens and dozens of services to monitor for really rare things. It should be part of general purpose monitoring suite.

Post reply on HN