Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Heroku deleted my database with no warning

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 216 posts

Re: Tell HN: Heroku deleted my database with no warning

#121

Post by hot_gril »

Andys wrote: post #118
hot_gril wrote: post #85

A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CA…

Yes, AWS is similar with their DB offerings. You can discourage it from doing any updates/reboots (which causes a failover), but ultimately if they want to failover, they can at any time.

Ouch. If it knows it's about to fail over from an update, it really should get the follower totally in sync with the leader first.

Re: Tell HN: Heroku deleted my database with no warning

#122

Post by tuukkah »

johannes1234321 wrote: post #109
robryan wrote: post #88

Earlier quoted context omitted.

Yeah ideally hold onto a backup for say a year, if the owner hasn't come and downloaded it after a year can then assume that they don't want it.

It's not that easy. Those databases might contain personal information which is protected under different privacy laws like GDPR, HIPAA and others and Heroku/Salesforce can not simply store that for longer than agreed upon and Heroku cancelling the account enables the retention period as the customer agreed upon as part of the T&C.

The account was not cancelled.

Re: Tell HN: Heroku deleted my database with no warning

#123

Post by hot_gril »

londons_explore wrote: post #105
hot_gril wrote: post #85

A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CA…

I wonder what architecture they use that can lose an hour of data? Most architectures I see might lose a few milliseconds of writes in the typical case, and perhaps a second of writes in the worst case (which occurs when the master gets islanded with a couple of clients).

If it was really the HA causing this, maybe the follower had a temp outage before the leader and hadn't yet caught up.

I already don't want HA if there's a chance for even 1 second of data loss, but for those who can tolerate that, there really should be an upper bound on the staleness. If your leader fails, the follower shouldn't take over unless it knows it's close to up-to-date.

Re: Tell HN: Heroku deleted my database with no warning

#124

Post by tevon »

The reputation hit to Heroku is real. Its so too bad that they've made these changes, and stopped pushing new features.

I built my prior business on heroku and it was wonderful. Now with this current company I've switched to Render (through am looking for something else). For me the writing was on the wall for Heroku and it wasn't worth it to me to dig in there if new features and support wasn't a given.

Curious for anyone with an inside perspective. What happened? Heroku was so far ahead of the pack for awhile, then suddenly stopped staying a step ahead. Was this intentional? Why?

Re: Tell HN: Heroku deleted my database with no warning

#125

Post by shkkmo »

tptacek wrote: post #114
shkkmo wrote: post #112

Earlier quoted context omitted.

The customer account wasn't terminated, the free DB being used by paid Dynos was deleted without any input from or notification to the customer. I highly doubt normal customers want this clock short when the cancellation is not customer initiated.

Data retention policies are written by compliance people, not product people, so distinctions like overt, deliberate cancellation and "cancellation for nonpayment" or "abandonment" or "discontinuation" usually aren't captured in them. I'm not saying it's great that Heroku deleted these databases; I'm saying: the description given upthread, that the databases were deleted because of contractual requirements, is super…

Plausible, sure, but that doesn't mean it isn't a cop out. There are many things that Heroku could have done to prevent this, like delaying the disconnection when they don't have confirmed delivery of the notifications and/or when they are connected to paid dynos/accounts.

Thus didn't happen because of the contract, but because the people implementing this transition didn't give a crap.

Re: Tell HN: Heroku deleted my database with no warning

#126

Post by tevon »

cpursley wrote: post #58
amenghra wrote: post #39

Earlier quoted context omitted.

I used Heroku’s free tier for small projects/prototype. I continue to use their cheapest offering for new projects because it saves me time. If a project grows, I can reevaluate my choice. I don’t think anything in Heroku is a pain to migrate away from. I would miss the intuitive/familiar UI.

Check out render.com, it's a solid alternative.

We currently use render. Its excellent, but is missing a couple key features: like Heroku's continuous rollback protection. If they offered this they'd be head and shoulders better.

Re: Tell HN: Heroku deleted my database with no warning

#127

Post by hot_gril »

fishnchips wrote: post #119
hot_gril wrote: post #101

Earlier quoted context omitted.

It's not exactly cheap to always have a backup that you know works. You have to set it up and test it periodically. Of course, that doesn't mean you shouldn't do it.

Very much that. As a former SRE responsible for tape storage, I saw things like regular backups of an error phrase "You have no access to this database". Guess what happened when the team accidentally dropped the database? Unless you're doing regular restores, you don't have a backup. You have hope. So yes, doing backups in a way that gives you some form of guarantee isn't exactly cheap.

Or we do have everything in the backup, but the restore process isn't worked out. Someone is losing a weekend writing hacky scripts, and every SLO is being violated, if we ever have to use it for real.

Re: Tell HN: Heroku deleted my database with no warning

#128

Post by yuubi »

CoastalCoder wrote: post #120
craigkerstiens wrote: post #29

Earlier quoted context omitted.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

> Even taking a dump and emailing it to me feels like a safer option here. I genuinely had to read this twice to get the intended meaning.

as someone who doesn't use heroku, so disregard my opinion:

i'd probably prefer feces-by-email to surprise database deletion

Re: Tell HN: Heroku deleted my database with no warning

#130

Post by codegeek »

kaushikc wrote: post #94
codegeek wrote: post #17

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even thoug…

Don't mind my curiosity but I wonder if you do offsite backup of your emails or just take it for granted that Gmail is a reliable enough service ? It is very important to me but I never even considered backing up my Google takeout data. The point I want to make is that service reliability is a critical factor when considering what to backup. It was expected from Heroku to do the right thing and be conservative in the…

I hear you. I didn't do it for a while but I did 2 things more recently:

1. Setup IMAP backups locally on my computer and everything is backed up on computer to 3rd party backup tool

2. Use Google Takeout from time to time (it is a bit weird at times though).

Post reply on HN