Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Soft deletion probably isn't worth it

brandur.org

Post reply on HN
Print view View original

371–380 of 514 posts

Re: Soft deletion probably isn't worth it

#371

Post by djur »

rodw wrote: post #303
djur wrote: post #296

Earlier quoted context omitted.

I would use "deleted_after" for the use case you describe (scheduled deletions). "deleted_at" is saying that an event occurred (deletion) at a specific time. There is never going to be a case where you have a deletion time for a record that is not deleted, nor should there ever be a time where a record is deleted but a time is not recorded. So a single timestamp column is a parsimonious solution.

I agree `deleted_after` is a more appropriate name than `deleted_at`, but it starts to run into naming conventions again. For example `published_at` is often used in editorial systems (newspapers, blogs, etc.) for things that "embargoed until" some future date. `published_after` or even just `pub_dt` would probably be more appropriate, but I feel like the `_at` suffix is well established in some contexts. For what it…

I have relied on *_at timestamps countless times, for support, analytics, etc. deleted_at in specific? Less often, sure.

Re: Soft deletion probably isn't worth it

#372

Post by JohnBooty »

eloisius wrote: post #364
EdwardDiego wrote: post #350

Earlier quoted context omitted.

I do a lot of work with Kafka, GPDR removal requests for data that's still stored in a Kafka topic is so much fun.

I worked at a large company as it was implementing GDPR compliance, and I can attest that it was a nightmare of drudgery. I think it's interesting that our entire tech stack is designed to assume the business has a right to keep user data for as long as they want (forever), and compliance with GDPR delete requests is like a one-off exception implemented at a very sketchy redaction task that has to be careful not to l…

Barring more foundational and revolutionary changes in our tech stacks, another poster commented that a best practice is to have tests that delete user accounts and all associated records without breakage and to build these tests into your test suite from the very inception of the project.

Far from a magic cure, but by building it into the app from inception, it seems more possible.

Re: Soft deletion probably isn't worth it

#373

Post by JohnBooty »

avianlyric wrote: post #282
smolder wrote: post #243

Earlier quoted context omitted.

I wonder what percentage of companies who even appear to comply with deletion requests actually do full deletion in practice. I suspect it's small, knowing how many things are coded to fake-delete for convenience. Businesses also tend to keep cold data backups around. (Maybe backups are exempted? I don't know.) There might even be cases where ostensibly deleted data can still be recovered from a disk, if they haven't…

Guess we’ll find out when a leaks happen, and companies start getting fines. GDPR does differentiate between structured data (I believe it uses the term “identifiable records” or similar), and huge heap of unstructured data where an individuals data can’t be quickly retrieved as it’s own atomic unit. With much stricter requirements for anything structured. So for data on a HDD that could be recovered, but is an unstr…

What does GDPR say about obscuring data?

Instead of hard-deleting a structured record for Bob Smith, can you leave the record intact and scrub it of identifying data so that I don't e.g. break all of the records for orders that Bob Smith made?

Re: Soft deletion probably isn't worth it

#374

Post by unemployable »

Everybody always did soft deletes with the is_deleted column at companies I once worked for so that is what I would do. I noticed that a lot of bugs would occur this way because you would forget to add the is_deleted to the query somewhere. The queries were also longer due to the longer where clause and so on.

These days I use a deleted table as per the article as I decided it would be better to deal with the more complex undelete process. It keeps that process to a single section instead of spreading it all throughout your database.

Some of the suggestions here like "use views" don't really work for two reasons - sometimes the is_deleted check must be performed in the ON clause, not in the WHERE clause, and sometimes you want to count the deleted or show the deleted, while other times you don't.

Re: Soft deletion probably isn't worth it

#375

Post by kleebeesh »

Maybe a more accurate take: Half-assed soft deletion definitely isn't worth it.

If you're just going to throw in some deleted bool or deleted_at timestamp without thorough testing, you might as well just skip it. It's virtually certain to go wrong.

Re: Soft deletion probably isn't worth it

#376

Post by 4m1rk »

JohnBooty wrote: post #151

I've been a software dev since the 90s and at this point, I've learned to basically do things like audit trails and soft deletion by default, unless there's some reason not to. Somebody always wants to undelete something, or examine it to see why it was deleted, or see who changed something, or blah blah blah. It helps the business, it helps you as developer by giving you debug information as well as helping you to c…

It's audit log enough for that? As deletion should also be logged.

Re: Soft deletion probably isn't worth it

#377

Post by gmiller123456 »

JohnBooty wrote: post #370
gmiller123456 wrote: post #172

Earlier quoted context omitted.

The author uses the "no one ever undeleted anything" as the primary justification. I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Either a complaint finally gets around to me as to why the user wanted their account deleted (e.g. feature not working) and it helps to figure out why. Or they're returning and want things set up l…

I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Yeah. As far as a user-facing "Undelete" button existing or being used... that's very rare in my experience. What's much more common is a user accidentally deletes some data. They deny they made an error. The developers are blamed. You then have to go on a wild goose chase figuri…

    How would you set up a secure audit trail that didn't rely on the application and/or database at some level?
A database is fine for an audit trail. But the application shouldn't have permissions to cover up a change. Likewise, the audit trail should record that a record was created, deleted, maybe undeleted, etc and when, by who. Relying only on the "deleted" flag only shows you the current state, not what and when was done and who did it.

Re: Soft deletion probably isn't worth it

#378

Post by [deleted] »

agentultra wrote: post #212

Soft deletion by storing the row in JSON won’t survive months of schema migrations. If restoring a record is rare you don’t want to have to find out that there’s no way to map the old data to the new table when it matters. There are cases where you shouldn’t be deleting or updating data; auditable and non-repudiation systems for some regulatory compliance come to mind. Best to use patterns that don’t require those op…

[deleted]

Re: Soft deletion probably isn't worth it

#379

Post by nikanj »

”Here’s my argument for why airbags are useless: In my 15 years of driving, I haven’t needed them once”

Remember the Attlassian outage from earlier this year. They sure would have appreciated a soft delete

Re: Soft deletion probably isn't worth it

#380

Post by unemployable »

Gurgler wrote: post #330

There's a very legitimate case that I've seen made for soft-deletion in several different situations: foreign keys related to "created-by" columns. Hard-deleting a user who created an object that remains in use after they're gone would trigger referential integrity complaints on those columns. Without being able to reference a "deactivated" user's primary key in such a situation, you'd have to come up with some count…

Yeah that is the main problem with not using soft deletes. The question is though, if you delete a user, should the user's personal information still exist in your database, or does that violate some kind of privacy regulations? The idea of the deleted user's table is that it can be kept around and then pruned after x number of days to satisfy both privacy and undeleting. To keep the references around, I think one way might be to create two tables, so one table is used for all of the references and it stays around, and the other one gets deleted. Eg Account and User tables, or something.
Post reply on HN