Live data from Hacker News

Tell HN: Heroku deleted my database with no warning

news.ycombinator.com

71–80 of 216 posts

Re: Tell HN: Heroku deleted my database with no warning

#71
post #23

Earlier quoted context omitted.

Did they explain why the notifications didn't work?

"I discussed this with our backend team and upon deeper investigation, we were able to confirm that there was some issue sending the notifications to your email." Nope.

Guesses 1, 2 and 3 are : their MTA's IP was black listed by a reputable mail hosting provider's spam filtering system (because they send spam).

Re: Tell HN: Heroku deleted my database with no warning

#72

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even thoug…

I spoke out loudly at a previous employer how it's really dumb to have our database and it's backups under a single AWS account. A single AWS account compromise, account issue (e.g. AWS shuts down account), or a disgruntled employee could result in the business being destroyed.

They took some half-hearted efforts to back up the data, but it was far from ideal.

Back up your data, and do it in different places so a failure in one won't affect any other copy of the data.

Re: Tell HN: Heroku deleted my database with no warning

#73
On heroku be sure you enable database backups with heroku pg:backups:schedule, and also script syncing those backups on a regular bases to some "off site" (non heroku) storage like your own s3 buckets. It is easy to get the URLs to you backups with `heroku pg:backups:url`.

I have no faith that if a rouge employee clicks the "delete app" button (because did you know to give an employee the ability to update the ssl cert on your web app you also need to give them permission to delete the whole damn app?) you'll ever be able to get your database back (although you might have a 30 day window to do so, but I wouldn't trust it.)

Re: Tell HN: Heroku deleted my database with no warning

#74

Earlier quoted context omitted.

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

Are you allowed to inform paying customers that you are going to do this? This is my primary complaint here. I don't understand how this oversight happened. This is going to cause an enormous amount of time and energy to recover from this.

I use heroku at work at they definitely sent out a ton of emails. It also said they were going to delete them right above all third party plugins (even if you weren’t using their database service)

Re: Tell HN: Heroku deleted my database with no warning

#75
post #37

For reference here's the email another user received in November. "Remind HN: Heroku will delete all free dbs and shut down all free dynos Monday" https://news.ycombinator.com/item?id=33755651

For me, the messaging in the announcements, dashboard, emails, and discussions here was clear and obvious that free databases were going to be deleted. If someone managed to miss all that, that's on them.

Re: Tell HN: Heroku deleted my database with no warning

#76
post #23

Earlier quoted context omitted.

Did they explain why the notifications didn't work?

"I discussed this with our backend team and upon deeper investigation, we were able to confirm that there was some issue sending the notifications to your email." Nope.

That phrasing doesn't rule out an issue on your email server's end, like the email could have bounced. (Though I'm guessing that's unlikely.)

Re: Tell HN: Heroku deleted my database with no warning

#77

Earlier quoted context omitted.

They deleted the database. On Heroku, databases and dynos are completely distinct entities, each with their own payment plans. When you set up a dyno and “add” a database, as most people do, it’s really easy to think that they are part of the same plan, and conclude that the “pro” or paid dyno is actually the combo of the dyno and database. They are not. I understand why they did this, databases can be shared between…

The least they could have done is disconnect the database without deleting the data, so users would at least have a chance to make the decision to either delete or start paying.

This is something I've put into most workflows at $(WORKSPACE=/=heroku) that end up disabling a customers systems. It depends a bit on how hard we have to kill a system for various factors. But for the softest setting, we usually start pulling DNS or similar access configs for a minute, wait for a minute, disable for 2, wait for 2, ... and the 30 minutes never end.

This gets people on the phone with support quickly if they don't react to their account manager or mails. Like, we've had a possible security incident coming from a customer system and they didn't react - because they tried to hide their processes internally. Scream-Testing got us on calls with their InfoSec in a hurry.

Re: Tell HN: Heroku deleted my database with no warning

#78
post #66

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even thoug…

After the Salesforce takeover? The one in 2010? So Heroku was good in like a year!?

Heh. I always loved Heroku and have complained that Salesforce is messing it up recently, but turns out the first time I used Heroku was like 5 years after the acquisition. I think Salesforce just started inserting their own branding more recently.

Re: Tell HN: Heroku deleted my database with no warning

#79

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…

Salesforce is like the Midas touch of destruction.

Re: Tell HN: Heroku deleted my database with no warning

#80

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…

render.com
Post reply on HN