Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Heroku deleted my database with no warning

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 216 posts

Re: Tell HN: Heroku deleted my database with no warning

#181

Post by Zababa »

codegeek wrote: post #17

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even thoug…

I wish this kind of warning would come with a link to a website like dobackups.org that would document how to actually do them and test them. Something like a front page with different "profiles" like "Windows user", "Web application", "Database". Add to that some more precise docs, links to different solutions, maybe a bit of transparent sponsorship from backup companies, good practices like "how often to do backups", more specialized info for different use cases like family photos, legal documents (with user-contributed infos by countries).

I would like to do backups but it's already not really easy, even harder when you add the whole "figure out how to do backups" to it.

Re: Tell HN: Heroku deleted my database with no warning

#182

Post by smsm42 »

tptacek wrote: post #103
craigkerstiens wrote: post #29

Earlier quoted context omitted.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

Having been through a SOC2 audit: this wouldn't fly. It's on the checklist of issues that you get hit with regardless of what kind of company you are: when customer accounts are terminated, the data retention clock starts ticking. You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short .…

Create a passphrase, derive the key, encrypt the data dump with the key, send the passphrase to the customer, delete the key. Presto, you can keep the data as long as you want without any privacy implications, since only the customer could decrypt it. Of course, this assumes working communication channel between the provider and the customer.

Re: Tell HN: Heroku deleted my database with no warning

#185

Post by dotancohen »

throwawaykai wrote: post #21
ericpauley wrote: post #4

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

The phrase "aren't allowed" supposes some regulatory agency forbidding an action. When it's your own internal policy that contradicts the action, the proper term is "won't".

Re: Tell HN: Heroku deleted my database with no warning

#186

Post by wombatpm »

justin_oaks wrote: post #72
codegeek wrote: post #17

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even thoug…

I spoke out loudly at a previous employer how it's really dumb to have our database and it's backups under a single AWS account. A single AWS account compromise, account issue (e.g. AWS shuts down account), or a disgruntled employee could result in the business being destroyed. They took some half-hearted efforts to back up the data, but it was far from ideal. Back up your data, and do it in different places so a fai…

Make them test their disaster recovery plan. One organization I worked for took three tries before they could even simulate disaster their recovery process.

Re: Tell HN: Heroku deleted my database with no warning

#190

Post by hyperman1 »

I am wondering more and more if any trust in a single cloud provider isn't simply an unacceptable risk. The power balance means that a minor error on their side is generally business-ending for you. This is different from almost any other supplier, which can generally be replaced, even if the business is on hold for a few days. So any data recovery plan that should be able answer: what if the relation with our cloud provider disappears unexpectedly.

This is not a warning against only heroku. Google is famous for terminating without any recourse random accounts because they felt like it that day. Amazon and especially Microsoft seem more dependable, but even they had their share of business-killing behaviour.

So e.g. a backup with another cloud vendor is a requirement for almost any business. And of course, validate it. Easyer said than done at scale, of course, but even a partially failed backup is better than nothing.

Post reply on HN