Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Soft deletion probably isn't worth it

brandur.org

Post reply on HN
Print view View original

431–440 of 514 posts

Re: Soft deletion probably isn't worth it

#431

Post by dcow »

heisjustsosmart wrote: post #408
coder543 wrote: post #401

Earlier quoted context omitted.

I feel like you’re overly dismissive of the other things I mentioned. Database failover comes with inconveniences, but it does allow you to use a single key and get great performance. For massive databases, it might be impossible to do that way, but most companies don’t have massive databases. I also opened a question about how ”erased” deleted data needs to be to be compliant, since I would imagine a lot of companie…

Save yourself a lot of time, read jandrewrogers comments before responding to him.

coder543 initially responded to dataflow. Then jandrewrogers included an additional anecdote, but only in a single comment. As the thread stands at the time I posed this, jandrewrogers does not have multiple comments.

Re: Soft deletion probably isn't worth it

#432

Post by qxxx »

in one project I was working on, we used a similar version of the 2nd method from the article:

Every table had the same table with _del suffix (eg. users_del). If a record was deleted, it was simply moved to _del table. We used code for this but later we started to use db triggers.

It worked quite well, and yes, there was always someone who wanted to undelete things. One downside was, if the schema changed on the source table, we needed to also change the schema in _del table. I like the approach with storing the data as json. That way there could be only 1 deleted_stuff table because it was looking quite strange having all the _del tables.

Re: Soft deletion probably isn't worth it

#433

Post by heisjustsosmart »

dcow wrote: post #431
heisjustsosmart wrote: post #408

Earlier quoted context omitted.

Save yourself a lot of time, read jandrewrogers comments before responding to him.

coder543 initially responded to dataflow. Then jandrewrogers included an additional anecdote, but only in a single comment. As the thread stands at the time I posed this, jandrewrogers does not have multiple comments.

An ambiguous phrase - I was not criticizing you but only advising you read his historical statements on databases before engaging. He has wasted a great deal of time of other people.

Re: Soft deletion probably isn't worth it

#434

Post by nicoburns »

dcow wrote: post #429
jandrewrogers wrote: post #389

Earlier quoted context omitted.

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor performance and scalability. Many classes of major optimization in databases don't work under these constraints. This isn't a novel idea, having been implemented for decades; it has been broadly rejected because it doesn't actually work in real systems without tra…

Is there literature on this topic? I'd like to learn at what scale per-user/tenant keys becomes untenable and the characteristics of systems that exhibit pathologically poor performance due to such a design. Naively, it sounds like individual entity keys does solve the problem of deletion, but that your argument is that the tradeoffs aren't generally worth it?

I’d imagine a pretty small scale. Imagine an endpoint that lists user accounts. You would have to individually decrypt each row! and you would never be able to do database joins of data in these tables because you it wouldn’t be possible to decrypt it until it had already left the database.

Re: Soft deletion probably isn't worth it

#435

Post by adrianmsmith »

bjourne wrote: post #57
munk-a wrote: post #24

I just wanted to touch on the fact that eliding soft-deleted rows from queries is really, really easy - this article makes it out to be a constant headache but here's my suggested approach. ALTER TABLE blah ADD COLUMN deleted_at NULL TIMESTAMP; ALTER TABLE blah RENAME TO blahwithdeleted; CREATE VIEW blah (SELECT * FROM blahwithdeleted WHERE deleted_at IS NULL); And thus your entire application just needs to keep SELE…

This is not a solution. It introduces a leaky abstraction which sooner or later will lead to errors. Sure, all code you write will access the view and not the table. But how can you ensure all other code in the organisation uses the view? Perhaps you add some access control to the table so that only authorized users can read directly from it, but that's even more technical overhead. Then you have foreign keys. If you…

> Someone new at the company makes a revenue report based in the billed Invoices and does not realize they should query the view and not the table... Not great if 90% of all invoices belong to soft-deleted customers!

This is not a great example of what you mean. If a customer purchased a product or subscription, paid for it, then later deleted their account, the company has received revenue so that data absolutely should be on revenue reports.

Re: Soft deletion probably isn't worth it

#437

Post by 411111111111111 »

brigandish wrote: post #417
411111111111111 wrote: post #404

Earlier quoted context omitted.

You literally said that soft deletes are illegal, which heavily implies what you now deny to have said. The 30 days wasn't particularly taken from the regulation, the actual grace period for the deletion should be 90 days in total, but I'm not a lawyer nor certain about it. What I am certain about is that your original thesis, which is soft deletes being illegal under gdpr, is complete bullshit

> You literally said that soft deletes are illegal, which heavily implies what you now deny to have said. Who is "you" here? Perhaps if you were less defensive you'd notice that I'm not the person you originally replied to, which, by the way, makes your response nonsensical. Edit: I'll add, that the only person who is likely to get anywhere near to "complete bullshit" is going to be the one who writes "but I'm not a…

Yeh, you take first place for the most retarded person 2022 award.

Re: Soft deletion probably isn't worth it

#438

Post by hnbad »

twelve40 wrote: post #268
alerighi wrote: post #216

Earlier quoted context omitted.

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

You can most definitely keep around the non-PII data you've generated for that user in your app, such as synthetic user ID's and other stuff. Furthermore, for some apps (fintech) in some countries (US) you are actually obligated to keep around certain information for several years. It really doesn't have to be all or nothing.

Synthetic user IDs can still qualify as PII since a human may still know which person that ID was associated with and data associated with that ID may be sufficient to narrow down who it was even if no human (or system) knows the exact mapping.

Pseudonymous data is not anonymous data.

But yes, the GDPR obviously allows for legal requirements of record keeping. It does however require you to delete the data once those requirements no longer apply (i.e. you have to ensure data is still deleted after those several years, you can't just not delete it because the deletion time is years in the future).

Re: Soft deletion probably isn't worth it

#440

Post by hnbad »

afiori wrote: post #253
alerighi wrote: post #216

Earlier quoted context omitted.

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

I would guess that there are quite a few limits to that... A user has a long history of participating on your forum and other users have quoted their messages far and wide. Collectively all of the messages posted on your forums (with or without timestamps) reveal a few PII about the user. Do you have to delete those? The user filed a bug report about a functionality not working, do you have to delete the text of the…

user_id is PII if the identity of a user can be narrowed down sufficiently.

For example, if you created a social media app and later left the company, it could still be determined that if you had an account it likely has one of the lowest user ID values if those were assigned sequentially. So even if your name was removed from it and none of your posts provided any identifying information about you, the user ID could still qualify as PII.

Likewise if the user ID is public and non-PII content associated with that ID is linked elsewhere identifying the author of that content (e.g. a news article embedding a post) the identity of the author is still compromised.

The problem is not just that users may submit PII in unexpected places, the problem is that even if the data type could not possibly allow them to, metadata beyond your control may actually still taint your supposedly anonymous data.

Post reply on HN