Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Soft deletion probably isn't worth it

brandur.org

Post reply on HN
Print view View original

401–410 of 514 posts

Re: Soft deletion probably isn't worth it

#401

Post by coder543 »

jandrewrogers wrote: post #389
coder543 wrote: post #365

Earlier quoted context omitted.

Remapping of blocks isn't the problem here. IANAL, but you should be encrypting all the data for GDPR compliance to begin with. With it encrypted, how the storage device chooses to map its blocks is completely irrelevant. When the key goes away, all the data is erased by definition . Throwing away the key is one of the easiest ways to comply with Right to be Forgotten, from what I've seen. "Oh, but the key is still o…

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor performance and scalability. Many classes of major optimization in databases don't work under these constraints. This isn't a novel idea, having been implemented for decades; it has been broadly rejected because it doesn't actually work in real systems without tra…

I feel like you’re overly dismissive of the other things I mentioned. Database failover comes with inconveniences, but it does allow you to use a single key and get great performance. For massive databases, it might be impossible to do that way, but most companies don’t have massive databases. I also opened a question about how ”erased” deleted data needs to be to be compliant, since I would imagine a lot of companies consider deletion on an encrypted drive to be good enough, even if the data may theoretically be recoverable under extreme circumstances until that hard drive is completely overwritten multiple times or destroyed.

Per user keys are also very useful for use cases like a personal file storage service, even if traditional RDBMS don’t work well with them. Techniques are situationally dependent.

I was not being dismissive of the existence of technical challenges, but rather dismissive of comments like:

> To be honest, in the age of modern overprovisioned storage drives that remap blocks frequently, I'm not really sure you can implement genuine "hard" deletes without choosing significantly unorthodox hardware (or destroying a drive every time you need a single bit erased), no matter how much you want to in software.

Which seem to completely deride the possibility of compliance.

Compliance is feasible, even if it has challenges.

Re: Soft deletion probably isn't worth it

#402

Post by jrumbut »

JohnBooty wrote: post #370
gmiller123456 wrote: post #172

Earlier quoted context omitted.

The author uses the "no one ever undeleted anything" as the primary justification. I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Either a complaint finally gets around to me as to why the user wanted their account deleted (e.g. feature not working) and it helps to figure out why. Or they're returning and want things set up l…

I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Yeah. As far as a user-facing "Undelete" button existing or being used... that's very rare in my experience. What's much more common is a user accidentally deletes some data. They deny they made an error. The developers are blamed. You then have to go on a wild goose chase figuri…

I've long wished for an RDBMS (or perhaps ORM, but I think it would be extra cool at the database level, see below) that does things that don't hyperscale.

So many LOB applications (the part of the iceberg under the water) have modestly sized databases with a relatively consistent number of users. The data is small, but often complex, awkwardly structured, highly relational, etc. The challenges these applications face are different, but real.

An example of something that would be incredibly useful here but completely untenable for a Facebook scale system would a rich rewind mechanism. You can go back in time 15 minutes or execute a query like:

RESTORE my_db TO LAST POINT WHERE EXISTS (SELECT * FROM ACCOUNTS WHERE ID=1234)

How much trouble would that kind of query have saved you in your career? At some point this would be cost prohibitive but I bet 50% or more of all apps never reach that point.

Re: Soft deletion probably isn't worth it

#403

Post by radu_floricica »

Is nobody using log tables? Pretty much every time I touch something in my db, there's a log call that records who did it, when, IP, URL and a (JSON) snapshot of the changed record, which in a pinch can be used for undelete.

It's surprisingly manageable. I mean, yes, it's definitely the largest table in the db, but:

1. it's well worth it

2. most of the stuff in it isn't the main scenario above (a human does something and I record the change) but various automated processes I also want to track, like API calls. which leads to:

3. it's easy to prune - both in time period kept, and by selectively deleting the automated stuff earlier

But it mostly helps by localizing things. It's just one meta-data log table, and everything related to logging actions is there. Not very elegant to keep adding fluff fields to every table, like "add_date" or "deleted_at". When I decided I want to also track the URL of the request I had to change things in just one place, and now I have it for every action everywhere.

Note: don't fall into the "everything is a nail" mistake. Some other dedicated log tables may be necessary, for high-volume or distinct stuff. I also have a mail_log, a sms_log and a separate table for events coming from mobile users (like location history).

Re: Soft deletion probably isn't worth it

#404

Post by 411111111111111 »

brigandish wrote: post #340
411111111111111 wrote: post #258

Earlier quoted context omitted.

Stop spreading misinformation. Nobody is required to hard delete the very second the [delete] button is pressed. You're following the law as long as that mentioned reaper job exists and runs at least once per month, to be safely within the allowed timeframe

I didn't see a claim that deletion had to be "the very second the [delete] button is pressed". This is the actual text of the law[1]: > The data subject shall have the right to obtain from the controller the erasure of personal data concerning him or her without undue delay There is a one month limit of the right of access [2] and right to be informed [3], which flow into some of the articles of the law; and certain…

You literally said that soft deletes are illegal, which heavily implies what you now deny to have said.

The 30 days wasn't particularly taken from the regulation, the actual grace period for the deletion should be 90 days in total, but I'm not a lawyer nor certain about it.

What I am certain about is that your original thesis, which is soft deletes being illegal under gdpr, is complete bullshit

Re: Soft deletion probably isn't worth it

#405

Post by salzig »

jrumbut wrote: post #402
JohnBooty wrote: post #370

Earlier quoted context omitted.

I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Yeah. As far as a user-facing "Undelete" button existing or being used... that's very rare in my experience. What's much more common is a user accidentally deletes some data. They deny they made an error. The developers are blamed. You then have to go on a wild goose chase figuri…

I've long wished for an RDBMS (or perhaps ORM, but I think it would be extra cool at the database level, see below) that does things that don't hyperscale. So many LOB applications (the part of the iceberg under the water) have modestly sized databases with a relatively consistent number of users. The data is small, but often complex, awkwardly structured, highly relational, etc. The challenges these applications fac…

CockroachDB has "AS OF SYSTEM TIME", which even allows you to backup the database for that moment.

https://www.cockroachlabs.com/docs/stable/as-of-system-time....

Re: Soft deletion probably isn't worth it

#406

Post by Groxx »

alerighi wrote: post #216
gmiller123456 wrote: post #172

Earlier quoted context omitted.

The author uses the "no one ever undeleted anything" as the primary justification. I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Either a complaint finally gets around to me as to why the user wanted their account deleted (e.g. feature not working) and it helps to figure out why. Or they're returning and want things set up l…

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

You can do both.

Soft delete immediately when requested. Hard delete (if still soft deleted) when GDPR requires it.

Besides it's not like all data is the same. Use what's appropriate when it's appropriate. Nobody's saying "you must always soft delete, and then immediately tweet the audit trail, print that tweet, put that printout onto a rocket, and launch it to the moon for later retrieval. with a blockchain for extra GDPR hate".

Re: Soft deletion probably isn't worth it

#407

Post by yuliyp »

avianlyric wrote: post #277
lazide wrote: post #274

Earlier quoted context omitted.

This should have the asterisk of * after the period of time you’ve said it will take for it to be deleted. It’s almost impossible to guarantee instant delete, folks just care if it’s gone in a documented and/or reasonable amount of time, and predictably. Correct?

Sure, but that time period is usually limited to 30 days, with an additional 30 day extension allowed if you contact the person and tell them you need more time. You can’t just declare that it take years to delete data, GDPR sets reasonable limits on how long a company can delay true deletion.

The GDPR says it's 30 days with a 2 month extension. This allows for having safe soft deletion and backup retention policies without running afoul of it.

Re: Soft deletion probably isn't worth it

#408

Post by heisjustsosmart »

coder543 wrote: post #401
jandrewrogers wrote: post #389

Earlier quoted context omitted.

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor performance and scalability. Many classes of major optimization in databases don't work under these constraints. This isn't a novel idea, having been implemented for decades; it has been broadly rejected because it doesn't actually work in real systems without tra…

I feel like you’re overly dismissive of the other things I mentioned. Database failover comes with inconveniences, but it does allow you to use a single key and get great performance. For massive databases, it might be impossible to do that way, but most companies don’t have massive databases. I also opened a question about how ”erased” deleted data needs to be to be compliant, since I would imagine a lot of companie…

Save yourself a lot of time, read jandrewrogers comments before responding to him.

Re: Soft deletion probably isn't worth it

#409

Post by gunnarmorling »

sverhagen wrote: post #400
gmiller123456 wrote: post #377

Earlier quoted context omitted.

How would you set up a secure audit trail that didn't rely on the application and/or database at some level? A database is fine for an audit trail. But the application shouldn't have permissions to cover up a change. Likewise, the audit trail should record that a record was created, deleted, maybe undeleted, etc and when, by who. Relying only on the "deleted" flag only shows you the current state, not what and when w…

Any tools you recommend here? Or just roll your own with triggers in the database?

Log-based change data capture is a great tool for building audit logs; unlike triggers, it doesn't impact write performance, you don't need to maintain triggers in the first place, and you can have you audit log in a system separate from your main database.

Blogged about one possible implementation using Debezium a while ago here: https://debezium.io/blog/2019/10/01/audit-logs-with-change-d...

Disclaimer: I work on Debezium

Re: Soft deletion probably isn't worth it

#410

Post by telotortium »

ikiris wrote: post #326
smolder wrote: post #243

Earlier quoted context omitted.

I wonder what percentage of companies who even appear to comply with deletion requests actually do full deletion in practice. I suspect it's small, knowing how many things are coded to fake-delete for convenience. Businesses also tend to keep cold data backups around. (Maybe backups are exempted? I don't know.) There might even be cases where ostensibly deleted data can still be recovered from a disk, if they haven't…

The really hard part is dealing with the data still in backups. It leads to having to do crazy things like individual keyed encryption per user, escrow hilarity, etc.

Or...just have a documented retention policy for backups and delete them after a certain timeline.
Post reply on HN