Soft deletion probably isn't worth it • Hacker News Board

Re: Soft deletion probably isn't worth it

#421

Post by xxs » Wed, Jul 20, 2022, 7:27 AM UTC

astura wrote: post #278

alerighi wrote: post #216

Earlier quoted context omitted.

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

>under the GDPR [soft delete] is illegal This is obviously not always true. Any European can't, for example, delete their online account with their mortgage company and demand that the mortgage company deletes the records saying that they owe them money so they can get their house for free. Nor can anyone call up their previous employer and require them to delete all the work they ever did from their company's comput…

Under GDPR the retention policy means the account has to be kept for whatever time stipulated by the law (10y), but then the data should not be retained any longer, hence deleted.

Re: Soft deletion probably isn't worth it

#422

Post by doctor_eval » Wed, Jul 20, 2022, 7:32 AM UTC

dubswithus wrote: post #344

doctor_eval wrote: post #260

Earlier quoted context omitted.

> the database is part of the application 100% this. If you accept that the database is part of the application, you give yourself permission to use the full feature set of the database, and life becomes a lot simpler. Using views, stored procedures and other features lets you implement things like soft delete trivially, without it infecting all your application code. In my entire career I've changed backend database…

> Using views, stored procedures and other features lets you implement things like soft delete trivially, without it infecting all your application code. That’s great but some of us actually like to write code. Especially Ruby on Rails where soft delete is a breeze if you don’t overthink it and build something the business doesn’t need.

Well, less code means less bugs, but go nuts. I'm just saying that the database is part of the stack. You wouldn't avoid Ruby features "just in case we stop using Ruby" - so why would we avoid using database features? It's up to you how best to assemble the features from your stack.

There is a reason we avoided database features in the 90's, which was to avoid database vendor lock-in. This was almost entirely a financial decision - you didn't want to be at the mercy of a vendor who could 10x the license fees and put you out of business, so you needed to have leverage over the vendor, and you needed to be able to credibly say you could change databases -- and in fact this exact scenario actually happened to me.

But that kind of behaviour is not really needed any more, and there's no need to avoid the great database features that systems like PG provide. In my experience, using database primitives to implement features tends to perform much faster (10x - 100x) and more consistently than the equivalent implementations higher up the stack.

Re: Soft deletion probably isn't worth it

#423

Post by doctor_eval » Wed, Jul 20, 2022, 7:34 AM UTC

Aeolun wrote: post #385

doctor_eval wrote: post #255

Earlier quoted context omitted.

File this under "falsehoods programmers believe about users": they act rationally.

Delete “I’m sorry sir, I didn’t think that clicking that button would actually delete something!” “What exactly did you think that button would do?”

"falsehoods programmers believe about users #2": they read button labels and dialog box text.

Re: Soft deletion probably isn't worth it

#424

Post by mmmuhd » Wed, Jul 20, 2022, 7:38 AM UTC

I remember when a rouge employee of a client went ahead to do stupid deletions on students' and staff data, soft delete saved the day and made us some money.

Re: Soft deletion probably isn't worth it

#425

Post by pwm » Wed, Jul 20, 2022, 7:41 AM UTC

jesseryoung wrote: post #283

JohnBooty wrote: post #151

I've been a software dev since the 90s and at this point, I've learned to basically do things like audit trails and soft deletion by default, unless there's some reason not to. Somebody always wants to undelete something, or examine it to see why it was deleted, or see who changed something, or blah blah blah. It helps the business, it helps you as developer by giving you debug information as well as helping you to c…

I was getting ready to disagree with you - but then I tried to think of any time I've actually pushed code to production with the "DELETE" keyword in it. The problems that I've had to solve in my career very rarely call for deleting something. "Soft deletion" and "audit trail" are technical terms we developers come up for solutions the business wants but maybe hasn't asked for yet. It's not really a soft deletion it'…

> Likewise, it's not an audit trail it's a "history" or "undo".

Depends on the industry. The one I work in audit trail is a well-defined and mandatory business concern.

Re: Soft deletion probably isn't worth it

#426

Post by dgb23 » Wed, Jul 20, 2022, 7:52 AM UTC

rjbwork wrote: post #175

augustl wrote: post #164

Earlier quoted context omitted.

This is why I don't understand why Datomic isn't more popular. Pretty much every system I've worked on never needed to scale past 100s of writes per second due to hard limits on the system (internal backoffice stuff, fundamenally scoped/shardable to defined regions, etc etc). And since Datomic is built with that in mind, you get the trade-off of full history, first class transactions and being able to query for thing…

In the past I've used MSSQL's Temporal Tables (also called System-Versioned Tables) to implement this kind of functionality. This also gets you, for free, Type 2 SCD functionality for OLAP-style queries. I can't wait until Postgres has this kind of functionality baked in. It's such a nice feature.

You likely know that there is an extension for PG that does this.

Re: Soft deletion probably isn't worth it

#427

Post by dcow » Wed, Jul 20, 2022, 7:56 AM UTC

JohnBooty wrote: post #151

I've been a software dev since the 90s and at this point, I've learned to basically do things like audit trails and soft deletion by default, unless there's some reason not to. Somebody always wants to undelete something, or examine it to see why it was deleted, or see who changed something, or blah blah blah. It helps the business, it helps you as developer by giving you debug information as well as helping you to c…

What's wrong with the author's proposed compromise/solution?

Re: Soft deletion probably isn't worth it

#428

Post by dgb23 » Wed, Jul 20, 2022, 7:58 AM UTC

JohnBooty wrote: post #319

corrral wrote: post #167

Earlier quoted context omitted.

I'd add tagging, for anything that could conceivably use it, when you're doing DB design. May as well start with support, even if the functionality's initially dormant. Someone will ask for it, directly or indirectly, and it won't take long before they do.

I personally would not agree. But... your experiences are also real and I believe you when you say that your experiences DO support your conclusion. :) The "soft delete" design decision is usually pretty impactful and is in my experience potentially much more of a pain in the butt to implement later if you haven't included it from day 0. Audit trails and soft-deletes are also crazy useful for developers (both for deb…

I agree. But if you’re asked to implement hierarchical categories, you implement them as tags underneath. This doesn’t cost you any time. When eventually they realize that categories don’t work (they never do) you can just turn a switch.

Re: Soft deletion probably isn't worth it

#429

Post by dcow » Wed, Jul 20, 2022, 8:09 AM UTC

jandrewrogers wrote: post #389

coder543 wrote: post #365

Earlier quoted context omitted.

Remapping of blocks isn't the problem here. IANAL, but you should be encrypting all the data for GDPR compliance to begin with. With it encrypted, how the storage device chooses to map its blocks is completely irrelevant. When the key goes away, all the data is erased by definition . Throwing away the key is one of the easiest ways to comply with Right to be Forgotten, from what I've seen. "Oh, but the key is still o…

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor performance and scalability. Many classes of major optimization in databases don't work under these constraints. This isn't a novel idea, having been implemented for decades; it has been broadly rejected because it doesn't actually work in real systems without tra…

Is there literature on this topic? I'd like to learn at what scale per-user/tenant keys becomes untenable and the characteristics of systems that exhibit pathologically poor performance due to such a design.

Naively, it sounds like individual entity keys does solve the problem of deletion, but that your argument is that the tradeoffs aren't generally worth it?

Re: Soft deletion probably isn't worth it

#430

Post by necovek » Wed, Jul 20, 2022, 8:10 AM UTC

taeric wrote: post #287

necovek wrote: post #267

Earlier quoted context omitted.

Exposing details of long lived background processes, and especially deletion processes to users (who in many cases couldn't care less) is a lot of work that's probably not worth it for the rare "hey, something went wrong" case — it's usually perfectly acceptable to raise that with the support team and let them use internal tools to debug and investigate. I am sure there are cases where it is worthwhile, but most of t…

I disagree. Any system that doesn't transition an entity to "deleting" is almost certainly going to be a pain point for me when I go to delete something. This can be seen as related to soft deletes. But I consider it more marking intent in the system. Lets you make "delete" a simple field update that will be carried out by the backend.

Say you are attempting to delete a tweet you made: you click on the delete button, and you want to see that your tweet is in the process of being deleted, but not really deleted yet (as the platform goes and updates all the retweet references and such)?

Really? What value do you as a user get from knowing the details of this transitory state? Do you also want to see a progress bar of how many references have been updated, what's the progress of evicting the tweet from search indexes, etc? What's wrong with all this appearing instantaneous and actual delete happening in the background over 5 minutes or 5 hours?

It seems you are in favour of this approach, just don't want to call it "soft delete" (since it may be followed by a hard out-of-band delete).