Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Heroku deleted my database with no warning

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 216 posts

Re: Tell HN: Heroku deleted my database with no warning

#161

Post by mst »

bryanrasmussen wrote: post #61

reading this I keep thinking the real questions are: 1. probably the notice sending and db deletion is two separate teams or responsibilities. Are they? 2. Did people know there was a bug in notice sending so some notices were not being sent, all notices not being sent? I ask this because generally in places I've worked where notice sending was an important part of things you knew if there was a bug and notices were…

My guess would be that a tiny percentage of accounts didn't get their notices sent, nobody complained about it because they didn't realise they were supposed to be getting them (or at least for whatever reason no such complaints were escalated to the people who would go "oh shit" upon seeing one) and some unfortunate combination of technical factors meant whatever internal monitoring existed didn't pick up the omissions.

Note that this is not to say the end result wasn't an indefensible disaster, only that disasters seldom have only a single cause and the above is my best uneducated guess at how things came together to cause this one.

Re: Tell HN: Heroku deleted my database with no warning

#162

Post by reiderrider »

Similar issue with billing post the Salesforce acquisition. We’re on a Enterprise Heroku plan but the billing could only be handled within a Salesforce account (which we don’t have). Autopay disconnected and our app went offline for 3 hours until we could confirm payment over the phone. Billing went from self serve to having to call in to provide a credit card over the phone.

Re: Tell HN: Heroku deleted my database with no warning

#163

Post by hot_gril »

mst wrote: post #158
hot_gril wrote: post #78

Earlier quoted context omitted.

Heh. I always loved Heroku and have complained that Salesforce is messing it up recently, but turns out the first time I used Heroku was like 5 years after the acquisition. I think Salesforce just started inserting their own branding more recently.

At a guess, Salesforce bought it hoping it would take over the entire space rather than merely having the capacity to be one profitable provider among many, and lost interest once it became clear it was going to be the latter rather than the former. (guess based only on having watched things unfold from the outside without ever being a user/customer so please do add salt to taste)

As a customer with no knowledge of the inside, this is all I've noticed, which probably misses some things:

1. GitHub integration breach, which wasn't handled well but luckily didn't affect me.

2. No more free-tier DBs :(

3. Salesforce logo, which ofc doesn't matter.

Hearing vague negative things here makes me nervous about the future because I really like Heroku and don't want to be stuck using AWS directly.

Re: Tell HN: Heroku deleted my database with no warning

#164

Post by nimchimpsky »

throwawaykai wrote: post #21
ericpauley wrote: post #4

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

[dead]

Re: Tell HN: Heroku deleted my database with no warning

#166

Post by kuon »

Heroku was so nice "back in the day", it was expensive but served me well. I started having some issues before the acquisition so I moved away. I guess I was lucky for that.

They must have had some billing problem because I got charged for months after stopping everything, I was able to block the charges at the credit card level but they kept trying and I never had any answer nor explanation from them.

Re: Tell HN: Heroku deleted my database with no warning

#167

Post by FPGAhacker »

craigkerstiens wrote: post #29
throwawaykai wrote: post #21

Earlier quoted context omitted.

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

There is a legal difference between a company policy and a contract with a customer.

Re: Tell HN: Heroku deleted my database with no warning

#168

Post by irjustin »

pas wrote: post #150
tptacek wrote: post #149

Earlier quoted context omitted.

Not so much, no.

If compliance affects any user-facing property of the product, then they are, by definition. Onboarding users/partners would be easier without making them read/click all those consent checkboxes, etc.

The simplest test is "who are you defending"?

Ideally, product people defend the end users. Realistically, compliance people defend the company from getting screwed in an audit and ultimately sued by the government.

Re: Tell HN: Heroku deleted my database with no warning

#169

Post by skissane »

throwawaykai wrote: post #21
ericpauley wrote: post #4

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at nig…

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

> We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

Contracts with some customers, surely? You could have the default be 90+ days, then those customers whose contracts specify a shorter timeframe get that shorter timeframe configured on their account instead. You could give the customer the choice at signup, and let them change it later using the settings console. If their contract doesn't specify a period, send them a notification that you will be changing it to 90+ days, but telling them they have the right to object if they disagree with that.

Re: Tell HN: Heroku deleted my database with no warning

#170

Post by arcticfox »

tptacek wrote: post #103
craigkerstiens wrote: post #29

Earlier quoted context omitted.

Having worked at Heroku and had a large part in building Heroku Postgres I do not recall this explicit policy, and it seems very squirrelly to me. Maybe this came in as a policy in recent years and it is the case, but still seems like hiding behind a policy as opposed to doing right by customers. You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and y…

Having been through a SOC2 audit: this wouldn't fly. It's on the checklist of issues that you get hit with regardless of what kind of company you are: when customer accounts are terminated, the data retention clock starts ticking. You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short .…

I don't think there's anything in SOC2 that says you can't have multiple tiers of deactivating an account. I think the crucial point here is that Salesforce is requesting the termination, not the customer, so any of their obligations to be responsive to customer data deletion requests are moot (unless the customer comes back and actually requests termination).
Post reply on HN