Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Soft deletion probably isn't worth it

brandur.org

Post reply on HN
Print view View original

501–510 of 514 posts

Re: Soft deletion probably isn't worth it

#501

Post by Falkon1313 »

quickthrower2 wrote: post #210
coding123 wrote: post #162

Earlier quoted context omitted.

I was going to chime in with this. thanks. One issue with views however is that a lot of these features require more and more nuanced knowledge of RDBMSes where these days unless you have a veteran architect, most of the team just knows the various library/tooling that interacts with "a variety of databases" so there is often less effort to go deeper.

Where is this anti-fb culture? Is it a startup thing? Everywhere I have worked people know a decent amount about their data store. Not architects, just mid devs and higher.

I've seen so much torturously complex code to coerce an ORM or query builder to generate a query that would've been simple just to write. Because "that's the way it's done" - Spend 5 minutes writing a query and then 5+ hours trying to figure out the convoluted code necessary to coerce something to generate that same query.

Of course, there are times when query builders are necessary to create the queries dynamically depending on parameters. But it's all too common to see them used and abused even for static queries.

People are just plain afraid of writing SQL. Or they think it's 'best practice' to use an ORM or query builder and using actual SQL is somehow 'wrong'.

Even people who know a decent amount about the database do it anyway because "that's the way that it's supposed to be done" or "that's 'best practice'".

Re: Soft deletion probably isn't worth it

#502

Post by necovek »

taeric wrote: post #477
necovek wrote: post #430

Earlier quoted context omitted.

Say you are attempting to delete a tweet you made: you click on the delete button, and you want to see that your tweet is in the process of being deleted, but not really deleted yet (as the platform goes and updates all the retweet references and such)? Really? What value do you as a user get from knowing the details of this transitory state? Do you also want to see a progress bar of how many references have been upd…

I'd rather see acknowledgement that it is having to do more work, than be surprised when it still shows up on another timeline for a bit longer. I don't necessarily care on a progress bar, as those typically have their own woes. And don't get me wrong, there are easy deletion cases that I am ok with appearing to happen immediately. I just find hiding processes from me is usually more annoying than it is worth. Worse,…

What you missed from my original post is that it is a lot of effort to expose transitory state: nobody is going the extra mile to annoy you, they are just not investing in implementing extra UI to expose it.

And in most cases, it's the right balance.

Re: Soft deletion probably isn't worth it

#503

Post by rodw »

dubswithus wrote: post #341
rodw wrote: post #275

Earlier quoted context omitted.

If any non-null value for deleted_at indicates a logical delete, it seems like TIMESTAMP is the wrong data type for that column. I mean I guess I can imagine a scenario where you want to know _when_ a record was logically deleted in addition to a flag that says is_deleted, but maybe that should be two different columns. Or at the very least, `deleted_at` is a misleading name in that case. Interpreting `deleted_at` as…

Every Rails developer knows what “deleted_at” means. I’m absolutely repulsed at the thought of turning the users table into a view. What’s next? No code and we only use triggers?

> I’m absolutely repulsed at the thought of turning the users table into a view. What’s next? No code and we only use triggers?

Assuming there's a justifiable need to apply the logical-delete idiom to the users table, what's the Rails-native, ActiveRecord-based approach to filtering out the logically deleted rows?

Re: Soft deletion probably isn't worth it

#504

Post by rodw »

d0100 wrote: post #309
rodw wrote: post #275

Earlier quoted context omitted.

If any non-null value for deleted_at indicates a logical delete, it seems like TIMESTAMP is the wrong data type for that column. I mean I guess I can imagine a scenario where you want to know _when_ a record was logically deleted in addition to a flag that says is_deleted, but maybe that should be two different columns. Or at the very least, `deleted_at` is a misleading name in that case. Interpreting `deleted_at` as…

> If any non-null value for deleted_at indicates a logical delete, it seems like TIMESTAMP is the wrong data type for that column. Not if I also want the timestamp of when it was deleted

That's fair. It's what I was trying to acknowledge when I wrote "I can imagine a scenario where you want to know _when_ a record was logically deleted".

That said, I'm genuinely curious about how often and in what way that time-based information is actually used.

For example, I often see created_at and modified_at columns that are completely ignored by the actual business logic and application code that are only used (if ever) for ad hoc diagnostic purposes like reconstructing the sequence of events after some catastrophic error in the app logic screwed up the database. If run-away-logic-screwing-up-the-database is a legitimate cause for concern, that's probably reason enough to include that extra meta-data, but I can easily count on one hand the number of times that's come up in practice across my long career. I mean, I also include that sort of column often. And there are certainly entities for which those data are relevant for application logic and/or reporting purposes. But there are definitely cases in which those data are never used at all too.

If this `deleted_at` timestamp-as-boolean convention is popular enough in the Rails community to be universally known, is there some common use for knowing _when_ a record was logically deleted (outside of reconstructing of the database state after something goes wrong) that I'm missing?

I.e., is this something that's actually used in the typical business logic and application code, or is it primarily used for ad hoc debugging?

To be clear I'm not _doubting_ that such a use case exists, it's just that off the top of my head I can't think of commonly occurring one.

Re: Soft deletion probably isn't worth it

#505

Post by alerighi »

dataflow wrote: post #248
alerighi wrote: post #216

Earlier quoted context omitted.

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

It's illegal but companies don't necessarily care to avoid soft deletes regardless. I think companies wait to get sued so they can try to argue in court why their soft deletions are reasonable and why it's too technically difficult for them to do hard deletes. To be honest, in the age of modern overprovisioned storage drives that remap blocks frequently, I'm not really sure you can implement genuine "hard" deletes wi…

> I think companies wait to get sued so they can try to argue in court why their soft deletions are reasonable and why it's too technically difficult for them to do hard deletes.

The law is clear, you can't keep user data if the user decides that you should no longer have them. It's your own responsibility to find a way to do that.

> To be honest, in the age of modern overprovisioned storage drives that remap blocks frequently, I'm not really sure you can implement genuine "hard" deletes without choosing significantly unorthodox hardware (or destroying a drive every time you need a single bit erased), no matter how much you want to in software. One of those details that I'm both surprised and unsurprised doesn't seem to have been addressed legally. I feel like a court ought to at least buy this aspect of the argument, so maybe they'll buy that it can be difficult in terms of the software too? Who knows. My guess is that a reasonable court would accommodate something that's reasonable for a given company, but there are lots of variations that could fall into that category.

The deletion should be done in reasonable terms, i.e. you can't reasonably get the data back. Of course when you delete something from an hard drive being that an SSD or HDD till that block is not reused the data is recoverable, but that is another thing.

By the way, it's either not really difficult to secure erase user data: what you need is not to erase all the data of the user, is to encrypt all the data of the user with a symmetric key algorithm (such as AES, that is super fast) and only store the key in a system that gives you secure erase capability. When the user deletes his account you don't have to delete all the data but just the encryption key.

Re: Soft deletion probably isn't worth it

#506

Post by alerighi »

JohnBooty wrote: post #373
avianlyric wrote: post #282

Earlier quoted context omitted.

Guess we’ll find out when a leaks happen, and companies start getting fines. GDPR does differentiate between structured data (I believe it uses the term “identifiable records” or similar), and huge heap of unstructured data where an individuals data can’t be quickly retrieved as it’s own atomic unit. With much stricter requirements for anything structured. So for data on a HDD that could be recovered, but is an unstr…

What does GDPR say about obscuring data? Instead of hard-deleting a structured record for Bob Smith, can you leave the record intact and scrub it of identifying data so that I don't e.g. break all of the records for orders that Bob Smith made?

You can as long as you delete all the personal data of the user and make sure that other remaining data is anonymized in a way that you can no longer associate it with the user.

Re: Soft deletion probably isn't worth it

#507

Post by alerighi »

afiori wrote: post #253
alerighi wrote: post #216

Earlier quoted context omitted.

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

I would guess that there are quite a few limits to that... A user has a long history of participating on your forum and other users have quoted their messages far and wide. Collectively all of the messages posted on your forums (with or without timestamps) reveal a few PII about the user. Do you have to delete those? The user filed a bug report about a functionality not working, do you have to delete the text of the…

> A user has a long history of participating on your forum and other users have quoted their messages far and wide. Collectively all of the messages posted on your forums (with or without timestamps) reveal a few PII about the user. Do you have to delete those?

Technically yes.

> The user filed a bug report about a functionality not working, do you have to delete the text of the bug report?

Depends of the privacy policy of the bug tracking application. If it's stated that the report shouldn't contain any personal data you don't have to delete it. If the user asks for deletion stating that it contain personal data you have to delete it.

> Arguably if your user table look like [user_id, creation_date, deletion_date, status, account_type] then this table does not contain any PII.

It doesn't, as long you no longer have any way to associate user_id to the user personal data, you can keep it.

> Sometimes users doxx themselves (like mistakenly sharing tax return forms instead cat pics), in such a case it is the user responsibility to signal this to you.

In that situation if the user asks for deletion of the content you have to provide it. You are obviously not asked to monitor errors about the user.

Re: Soft deletion probably isn't worth it

#508

Post by taeric »

necovek wrote: post #502
taeric wrote: post #477

Earlier quoted context omitted.

I'd rather see acknowledgement that it is having to do more work, than be surprised when it still shows up on another timeline for a bit longer. I don't necessarily care on a progress bar, as those typically have their own woes. And don't get me wrong, there are easy deletion cases that I am ok with appearing to happen immediately. I just find hiding processes from me is usually more annoying than it is worth. Worse,…

What you missed from my original post is that it is a lot of effort to expose transitory state: nobody is going the extra mile to annoy you, they are just not investing in implementing extra UI to expose it. And in most cases, it's the right balance.

From my experience, most places don't model the transition state. Such that it easily gets bunged.

If you just want to make it hidden, that is fine. But can lead to it's own problems. Usually when you have uniqueness constraints on things.

Re: Soft deletion probably isn't worth it

#509

Post by dataflow »

alerighi wrote: post #505
dataflow wrote: post #248

Earlier quoted context omitted.

It's illegal but companies don't necessarily care to avoid soft deletes regardless. I think companies wait to get sued so they can try to argue in court why their soft deletions are reasonable and why it's too technically difficult for them to do hard deletes. To be honest, in the age of modern overprovisioned storage drives that remap blocks frequently, I'm not really sure you can implement genuine "hard" deletes wi…

> I think companies wait to get sued so they can try to argue in court why their soft deletions are reasonable and why it's too technically difficult for them to do hard deletes. The law is clear, you can't keep user data if the user decides that you should no longer have them. It's your own responsibility to find a way to do that. > To be honest, in the age of modern overprovisioned storage drives that remap blocks…

These points have already been discussed, see the sibling threads.

Re: Soft deletion probably isn't worth it

#510

Post by likortera »

joshmanders wrote: post #231
rjzzleep wrote: post #170

Earlier quoted context omitted.

In rails you get these things for free. What I don't get is why everyone rolls their own framework with node.js. It's basically 90s PHP all over again. EDIT: Soft delete is a trivial piece of code when the framework has a well defined transaction system for its ORM. It's not really related to Rails per se. Your statement is extremely disingenuous, while trying to look smart. Audit trails _can_ be(but don't have to be…

I don't know why you're ragging on Node.js users or even PHP for that matter as both ecosystems have this stuff covered too. Also you're comparing language/runtime with an actual framework and then dogging those users... If you want to compare Rails with Node/PHP then I'd suggest comparing with things like Laravel (PHP), Adonis (Node) and you'll find everything you can do in Rails is done in Node/PHP too.

Adonis is far, far, far from reaching even a small percent of what Laravel is. It has almost no community and very little or almost no third party packages. When I tried it las year I couldn't even find a formatter for the templates.

Later I've found out the author of the framework has fake accounts pretending to be a happy user of it, and that was the end of it for me.

So, not comparable at all in my opinion.

Post reply on HN