Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Heroku deleted my database with no warning

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 216 posts

Re: Tell HN: Heroku deleted my database with no warning

#111

Post by johannes1234321 »

reaperducer wrote: post #91
joemi wrote: post #65

Earlier quoted context omitted.

So you're in favor of companies breaking their terms and conditions at will? I think that would cause quite a lot more outrage and problems.

So you're in favor of companies breaking their terms and conditions at will? I think that would cause quite a lot more outrage and problems. Do you really never get e-mails from various companies every few weeks telling you that the terms of service have been updated? I think I get one from eBay alone every other day.

Those can not be compared that easily. There are many changes you can do easily. Changing data retention periods impacts guarantees customers may have given in compliance to privacy regulation like GDPR, HIPAA etc. and thus is a substantial change to the customers in areas with such regulation.

Re: Tell HN: Heroku deleted my database with no warning

#112

Post by shkkmo »

tptacek wrote: post #103
craigkerstiens wrote: post #29

Earlier quoted context omitted.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

Having been through a SOC2 audit: this wouldn't fly. It's on the checklist of issues that you get hit with regardless of what kind of company you are: when customer accounts are terminated, the data retention clock starts ticking. You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short .…

The customer account wasn't terminated, the free DB being used by paid Dynos was deleted without any input from or notification to the customer.

I highly doubt normal customers want this clock short when the cancellation is not customer initiated.

Re: Tell HN: Heroku deleted my database with no warning

#113

Post by tacker2000 »

throwawaykai wrote: post #21
ericpauley wrote: post #4

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

[flagged]

Re: Tell HN: Heroku deleted my database with no warning

#114

Post by tptacek »

shkkmo wrote: post #112
tptacek wrote: post #103

Earlier quoted context omitted.

Having been through a SOC2 audit: this wouldn't fly. It's on the checklist of issues that you get hit with regardless of what kind of company you are: when customer accounts are terminated, the data retention clock starts ticking. You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short .…

The customer account wasn't terminated, the free DB being used by paid Dynos was deleted without any input from or notification to the customer. I highly doubt normal customers want this clock short when the cancellation is not customer initiated.

Data retention policies are written by compliance people, not product people, so distinctions like overt, deliberate cancellation and "cancellation for nonpayment" or "abandonment" or "discontinuation" usually aren't captured in them. I'm not saying it's great that Heroku deleted these databases; I'm saying: the description given upthread, that the databases were deleted because of contractual requirements, is super plausible.

Re: Tell HN: Heroku deleted my database with no warning

#115

Post by crote »

tptacek wrote: post #103
craigkerstiens wrote: post #29

Earlier quoted context omitted.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

Having been through a SOC2 audit: this wouldn't fly. It's on the checklist of issues that you get hit with regardless of what kind of company you are: when customer accounts are terminated, the data retention clock starts ticking. You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short .…

But this isn't an account termination. They deleted the data of an active account, without informing the user.

Why could they not turn it into a read-only database without access from the Heroku apps instead? Then it'd just be a routine change to the service offered, would it not?

Re: Tell HN: Heroku deleted my database with no warning

#116

Post by nineteen999 »

> I guess the bigger lesson though is that Heroku should really be a last resort option for projects these days.

Maybe people shouldn't, I don't know because I've never used it, but ...

> I learned my lesson about not doing an offsite backup regularly.

This is the bigger lesson, no matter whether you are using third party database hosting, or hosting yourself, whether it's a NoSQL database or a SQL one.

Once the data is gone, its gone.

Re: Tell HN: Heroku deleted my database with no warning

#117

Post by lamontcg »

codegeek wrote: post #17

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even thoug…

Backups are apparently "too 20th century" for today's cloud-focused devops, you can just trust your entire business to someone else's procedures and if it all disappears in a puff of smoke, so do you.

(or just maybe we shouldn't have thrown out all the sysadmins with the bathwater)

Re: Tell HN: Heroku deleted my database with no warning

#118

Post by Andys »

hot_gril wrote: post #85

A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CA…

Yes, AWS is similar with their DB offerings. You can discourage it from doing any updates/reboots (which causes a failover), but ultimately if they want to failover, they can at any time.

Re: Tell HN: Heroku deleted my database with no warning

#119

Post by fishnchips »

hot_gril wrote: post #101
robryan wrote: post #92

Earlier quoted context omitted.

Yeah pretty cheap and easy, at least relative to the value of data to most companies to setup a server you have physical control of to go download the backup once per day.

It's not exactly cheap to always have a backup that you know works. You have to set it up and test it periodically. Of course, that doesn't mean you shouldn't do it.

Very much that. As a former SRE responsible for tape storage, I saw things like regular backups of an error phrase "You have no access to this database". Guess what happened when the team accidentally dropped the database?

Unless you're doing regular restores, you don't have a backup. You have hope. So yes, doing backups in a way that gives you some form of guarantee isn't exactly cheap.

Re: Tell HN: Heroku deleted my database with no warning

#120

Post by CoastalCoder »

craigkerstiens wrote: post #29
throwawaykai wrote: post #21

Earlier quoted context omitted.

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

> Even taking a dump and emailing it to me feels like a safer option here.

I genuinely had to read this twice to get the intended meaning.

Post reply on HN