Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Heroku deleted my database with no warning

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 216 posts

Re: Tell HN: Heroku deleted my database with no warning

#101

Post by hot_gril »

robryan wrote: post #92
justin_oaks wrote: post #72

Earlier quoted context omitted.

I spoke out loudly at a previous employer how it's really dumb to have our database and it's backups under a single AWS account. A single AWS account compromise, account issue (e.g. AWS shuts down account), or a disgruntled employee could result in the business being destroyed. They took some half-hearted efforts to back up the data, but it was far from ideal. Back up your data, and do it in different places so a fai…

Yeah pretty cheap and easy, at least relative to the value of data to most companies to setup a server you have physical control of to go download the backup once per day.

It's not exactly cheap to always have a backup that you know works. You have to set it up and test it periodically. Of course, that doesn't mean you shouldn't do it.

Re: Tell HN: Heroku deleted my database with no warning

#102

Post by TimWolla »

kaushikc wrote: post #94
codegeek wrote: post #17

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even thoug…

Don't mind my curiosity but I wonder if you do offsite backup of your emails or just take it for granted that Gmail is a reliable enough service ? It is very important to me but I never even considered backing up my Google takeout data. The point I want to make is that service reliability is a critical factor when considering what to backup. It was expected from Heroku to do the right thing and be conservative in the…

Personally I sync the emails in my Gmail account to my server at Hetzner using imapsync [1] every once in a while. The data on that server is backed up off-site at rsync.net using Borg. I'm intentionally not using Hetzner's storage boxes, because I want my backup to be entirely independent, e.g. if my Hetzner customer account is closed down for whatever reason.

[1] https://imapsync.lamiral.info/

Re: Tell HN: Heroku deleted my database with no warning

#103

Post by tptacek »

craigkerstiens wrote: post #29
throwawaykai wrote: post #21

Earlier quoted context omitted.

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

Having been through a SOC2 audit: this wouldn't fly. It's on the checklist of issues that you get hit with regardless of what kind of company you are: when customer accounts are terminated, the data retention clock starts ticking.

You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short. They don't want you to retain their stuff after they cancel.

Re: Tell HN: Heroku deleted my database with no warning

#104

Post by fireworks »

lalopalota wrote: post #75
mtmail wrote: post #37

For reference here's the email another user received in November. "Remind HN: Heroku will delete all free dbs and shut down all free dynos Monday" https://news.ycombinator.com/item?id=33755651

For me, the messaging in the announcements, dashboard, emails, and discussions here was clear and obvious that free databases were going to be deleted. If someone managed to miss all that, that's on them.

No in-app notifications, no emails, and I was travelling and not following the news closely. I don't think I should have to rely on forums to know if I'm going to have all my data deleted without warning as a paying customer anyway.

Re: Tell HN: Heroku deleted my database with no warning

#105

Post by londons_explore »

hot_gril wrote: post #85

A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CA…

I wonder what architecture they use that can lose an hour of data?

Most architectures I see might lose a few milliseconds of writes in the typical case, and perhaps a second of writes in the worst case (which occurs when the master gets islanded with a couple of clients).

Re: Tell HN: Heroku deleted my database with no warning

#106

Post by eptcyka »

fireworks wrote: post #41
yamtaddle wrote: post #40

Earlier quoted context omitted.

> Are you allowed to inform paying customers that you are going to do this? I can't be the only one who's basically completely blind to emails from major companies, including SaaS providers, because they're so fucking spammy that the SNR is like 1:99. Notifying me by email, for one of these places, is functionally the same as not notifying me at all. [EDIT] Sorry, didn't mean to imply the parent wasn't paying attenti…

I'm not blind, they didn't send one. And they admitted to me that they did not send one.

That's a major failure on their part. I did get several nagging emails from them.

Re: Tell HN: Heroku deleted my database with no warning

#107

Post by brogrammernot »

glenngillen wrote: post #99
xp84 wrote: post #20

Earlier quoted context omitted.

I assumed the value prop of being on Heroku in 2023 was purely "not having to do the pain in the butt to migrate your legacy app that you made on Heroku in the pre-2010 era" - in other words, I would be really shocked if Heroku got significant new business or any growth at all, now that it's just expensive AWS with some basic CI integration points. I also assume Salesforce only bought them as a cash generator and has…

> Salesforce on bought them as a cash generator As an early Heroku employee, this gave me quite the chuckle.

Same - wasn’t an employee but used it often in ~2013-2015 timeframe. I’m pretty sure the valuation was many many many multiples of any sort of revenue based metric, wasn’t it?

Re: Tell HN: Heroku deleted my database with no warning

#108

Post by inopinatus »

throwawaykai wrote: post #21
ericpauley wrote: post #4

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

This sounds like the legal equivalent of looking the wrong way through a telescope.

Whoever fostered that naive interpretation was a nitwit. If they’re an actual lawyer, they promoted an intentional, mutually harmful unilateral reinterpretation of an agreement and should be sacked.

Cowering behind T&Cs like this is intellectual bankruptcy. There’s always another solution. The law is not a programming language.

Re: Tell HN: Heroku deleted my database with no warning

#109

Post by johannes1234321 »

robryan wrote: post #88
craigkerstiens wrote: post #29

Earlier quoted context omitted.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

Yeah ideally hold onto a backup for say a year, if the owner hasn't come and downloaded it after a year can then assume that they don't want it.

It's not that easy. Those databases might contain personal information which is protected under different privacy laws like GDPR, HIPAA and others and Heroku/Salesforce can not simply store that for longer than agreed upon and Heroku cancelling the account enables the retention period as the customer agreed upon as part of the T&C.
Post reply on HN