Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Soft deletion probably isn't worth it

brandur.org

Post reply on HN
Print view View original

381–390 of 514 posts

Re: Soft deletion probably isn't worth it

#381

Post by smolder »

ikiris wrote: post #326
smolder wrote: post #243

Earlier quoted context omitted.

I wonder what percentage of companies who even appear to comply with deletion requests actually do full deletion in practice. I suspect it's small, knowing how many things are coded to fake-delete for convenience. Businesses also tend to keep cold data backups around. (Maybe backups are exempted? I don't know.) There might even be cases where ostensibly deleted data can still be recovered from a disk, if they haven't…

The really hard part is dealing with the data still in backups. It leads to having to do crazy things like individual keyed encryption per user, escrow hilarity, etc.

Thanks for that insight. It's an interesting solution to use per-user encryption for that. Don't you have the same problems all over again with respect to preserving access to user specific data? I.e. needing key backups for business continuity, but needing to wipe those keys for deletion requests.

Re: Soft deletion probably isn't worth it

#382

Post by dataflow »

coder543 wrote: post #365
dataflow wrote: post #280

Earlier quoted context omitted.

I don't know if you were the exception or the rule, but I know for a fact some do go the forgiveness approach for requirements that they believe would be disproportionately burdensome. Probably almost all companies actually, including yourself (just to a different extent) - what did you do about the storage remapping thing I mentioned? Did it come up/did you guys discuss it? Do you believe you're in compliance despit…

Remapping of blocks isn't the problem here. IANAL, but you should be encrypting all the data for GDPR compliance to begin with. With it encrypted, how the storage device chooses to map its blocks is completely irrelevant. When the key goes away, all the data is erased by definition . Throwing away the key is one of the easiest ways to comply with Right to be Forgotten, from what I've seen. "Oh, but the key is still o…

I have no idea where the sudden hostility in your last paragraph came from, but it's ascribing nonexistent malice to me. I'll try to address a couple of your points the best I can regardless.

> Once a month, you copy the per-user keys that haven't been deleted onto a new encrypted volume, then tell the TPM to delete the key for the old encrypted volume: garbage collecting the deleted keys in an unrecoverable fashion. This avoids the hassles with having to failover a database so frequently

This ignores the existence of database indexes.

What I was saying (I thought quite explicitly) was that strict compliance to GDPR appears unreasonably burdensome to most companies, and thus most companies stop at a line they draw earlier (that line frequently being the lack of hardware guarantees re: block remapping). This is not the same as saying no one can comply with GDPR, and I had no intention of making GDPR look impossible to comply with. Quite the contrary in fact—I believe its current requirements can and should be complied with quite cheaply and efficiently, but the most effective path to that requires support from hardware manufacturers (and possibly others) that customers and regulators currently don't demand.

This is not a take-down of GDPR or a malicious strawman mockery of its requirements as you're (quite unkindly and wrongly) imputing me with. If anything, it was a plea for customers, regulators, and/or lawmakers to realize they need to expand their GDPR demands to require that vendors of other parts of the hardware/software stack (particularly storage device manufacturers) provide better support for GDPR compliance.

Re: Soft deletion probably isn't worth it

#383

Post by rvr_ »

TFA is nonsense. Hard deletes should almost never be used, period. The application credentials should not even have permission to issue delete statements, thus reducing potential damage from bad actors. Things like ON CASCADE DELETE should not even exist. Anyone using them must stop and rethink their life decisions.

Re: Soft deletion probably isn't worth it

#384

Post by [deleted] »

JohnBooty wrote: post #319
corrral wrote: post #167

Earlier quoted context omitted.

I'd add tagging, for anything that could conceivably use it, when you're doing DB design. May as well start with support, even if the functionality's initially dormant. Someone will ask for it, directly or indirectly, and it won't take long before they do.

I personally would not agree. But... your experiences are also real and I believe you when you say that your experiences DO support your conclusion. :) The "soft delete" design decision is usually pretty impactful and is in my experience potentially much more of a pain in the butt to implement later if you haven't included it from day 0. Audit trails and soft-deletes are also crazy useful for developers (both for deb…

[deleted]

Re: Soft deletion probably isn't worth it

#385

Post by Aeolun »

doctor_eval wrote: post #255
wst_ wrote: post #236

Earlier quoted context omitted.

Isn't it the problem of UI, though. If the user would be informed about the consequences (possibly with bold red font and with a confirmation checkbox) would they still click that button?

File this under "falsehoods programmers believe about users": they act rationally.

Delete

“I’m sorry sir, I didn’t think that clicking that button would actually delete something!”

“What exactly did you think that button would do?”

Re: Soft deletion probably isn't worth it

#386

Post by [deleted] »

gmiller123456 wrote: post #172
JohnBooty wrote: post #151

I've been a software dev since the 90s and at this point, I've learned to basically do things like audit trails and soft deletion by default, unless there's some reason not to. Somebody always wants to undelete something, or examine it to see why it was deleted, or see who changed something, or blah blah blah. It helps the business, it helps you as developer by giving you debug information as well as helping you to c…

The author uses the "no one ever undeleted anything" as the primary justification. I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Either a complaint finally gets around to me as to why the user wanted their account deleted (e.g. feature not working) and it helps to figure out why. Or they're returning and want things set up l…

[deleted]

Re: Soft deletion probably isn't worth it

#387

Post by petre »

alerighi wrote: post #216
gmiller123456 wrote: post #172

Earlier quoted context omitted.

The author uses the "no one ever undeleted anything" as the primary justification. I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Either a complaint finally gets around to me as to why the user wanted their account deleted (e.g. feature not working) and it helps to figure out why. Or they're returning and want things set up l…

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

We don't do it for users but for certain user assets that are a pain to remove consistently without a background job or a job queue. Like user devices that have associated dependencies. The background job is automatic, it prunes objects evey x hours.

Re: Soft deletion probably isn't worth it

#389

Post by jandrewrogers »

coder543 wrote: post #365
dataflow wrote: post #280

Earlier quoted context omitted.

I don't know if you were the exception or the rule, but I know for a fact some do go the forgiveness approach for requirements that they believe would be disproportionately burdensome. Probably almost all companies actually, including yourself (just to a different extent) - what did you do about the storage remapping thing I mentioned? Did it come up/did you guys discuss it? Do you believe you're in compliance despit…

Remapping of blocks isn't the problem here. IANAL, but you should be encrypting all the data for GDPR compliance to begin with. With it encrypted, how the storage device chooses to map its blocks is completely irrelevant. When the key goes away, all the data is erased by definition . Throwing away the key is one of the easiest ways to comply with Right to be Forgotten, from what I've seen. "Oh, but the key is still o…

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor performance and scalability. Many classes of major optimization in databases don't work under these constraints. This isn't a novel idea, having been implemented for decades; it has been broadly rejected because it doesn't actually work in real systems without tradeoffs no one accepts, even in highly security sensitive environments.

You are overly dismissive of the technical challenges of actually deleting data, particularly in large systems. Technically naive solutions like having separate encryption keys for individual entities sounds good but it doesn't actually address the problem.

Re: Soft deletion probably isn't worth it

#390

Post by mixmastamyk »

codemac wrote: post #42

Well, there are several problems with this analysis when you go very large (>10000 machines): - For many applications, it's easiest to put the state of the object in the primary key, and thus point reads will fail when something gets deleted. This has other problems though with hotspotting and compaction during deletes. The deleted table doesn't really solve this either. - For storage systems, GC is critical function…

Advice should be aimed at the 99% rather than the 1%, right? I guess Heroku and Stripe don’t have the biggest datasets in the world but they are probably larger than most folks will need to manage.
Post reply on HN