They were charging money for thing A, that connected to (free) thing B, and rather than just begin charging the same customer, using same credit card, for thing B, they deleted it. I would love to see the recording of that zoom meeting...
Tell HN: Heroku deleted my database with no warning
81–90 of 216 posts
Re: Tell HN: Heroku deleted my database with no warning
#82does seem like a cautionary story that should rule heroku out for anything except quick experiments
Re: Tell HN: Heroku deleted my database with no warning
#83I can just second most of those comments here.
One thing should be clear: It doesn't matter if you have a payed or free plan with someone who stores your data. Always make backups!
There are so many things what can go wrong
Re: Tell HN: Heroku deleted my database with no warning
#84You didn't have a full backup of your database? That's on you. Disaster Recovery is an important part of any project. Heroku could have blipped out of existence by accident and you would still have the same problem.
Re: Tell HN: Heroku deleted my database with no warning
#85A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CAP theorem). Today I have enough experience to check the fine print for that kind of detail, but anyway such a big change should come with big bold letters IMO.
[Edit: To this day I'm still puzzled by what I'm about to describe, so idk if it's Heroku's fault or mine.] I got a call from my colleague one day saying our database had gone back in time. Evidently we lost an hour of records. The code wasn't even capable of deleting rows, and nobody had direct DB access but me, so after leafing through the docs I suspected a failover event caused it. Premium DBs also let you roll back the DB to a previous point in time, and we were able to recover most of our data this way, like Back to the Future. If this really was a failover event, it's super weird if that the backup was more up to date than the standby master, and that a whole hour (rather than minute) was lost.
Re: Tell HN: Heroku deleted my database with no warning
#86Oh crap! What a horror story.
We recently migrated from Heroku Postgres to Crunchy Bridge and can totally recommend it. So maybe set up your new database there instead.
Re: Tell HN: Heroku deleted my database with no warning
#87I remember being quite surprised when I first learned that Heroku was owned by Salesforce, because I had such a different impression of the two companies. I haven't used it in a few years, but Heroku used to be a great platform for certain types of projects. Unfortunately, the number of concerning stories I have heard about them in recent years has discouraged me from ever using Heroku for a future project.
Re: Tell HN: Heroku deleted my database with no warning
#88Earlier quoted context omitted.
Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.
Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…
Yeah ideally hold onto a backup for say a year, if the owner hasn't come and downloaded it after a year can then assume that they don't want it.
Re: Tell HN: Heroku deleted my database with no warning
#89A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CA…
Having a HA follower is the only different between Premium and Standard tiers, so I'm not really sure what else you expected them to do in this case. Like, premium-6 is 2x the cost of the standard-6 plan explicitly because of the HA follower.
Re: Tell HN: Heroku deleted my database with no warning
#90This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…
Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.
If I'm understanding you correctly, the 30 day policy is one that Heroku chose to put in the contract. Engineering might have fought the terms, and yes they need to be followed once set, but it seems totally fair to blame Heroku for creating the limitation in the first place.