Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Soft deletion probably isn't worth it

brandur.org

Post reply on HN
Print view View original

441–450 of 514 posts

Re: Soft deletion probably isn't worth it

#441

Post by ReptileMan »

JohnBooty wrote: post #151

I've been a software dev since the 90s and at this point, I've learned to basically do things like audit trails and soft deletion by default, unless there's some reason not to. Somebody always wants to undelete something, or examine it to see why it was deleted, or see who changed something, or blah blah blah. It helps the business, it helps you as developer by giving you debug information as well as helping you to c…

The proper thing is database snapshots or something like that.

I don't think soft delete is wrong per se, but it is something that should be native to the database engine.

Re: Soft deletion probably isn't worth it

#442

Post by hnbad »

M2Ys4U wrote: post #334
he0001 wrote: post #313

Earlier quoted context omitted.

> Nor can anyone call up previous employer and require them to delete all the work they ever did from their company’s computer systems. You can’t delete IP as this is not covered by GDPR, but you sure can ask them to delete your identification data from their records as GDPR also works for employees not just customers. It’s a problem with for example vcs. Unless there’s another law in ruling, GDPR is the baseline. It…

>You can’t delete IP as this is not covered by GDPR, but you sure can ask them to delete your identification data from their records as GDPR also works for employees not just customers. You can ask , but the right to have data erased is not an absolute right. In fact there's an enumerated list of the circumstances in which the data must be deleted but data controllers are otherwise under no specific obligation to del…

An often missed case of the GDPR requiring companies to delete data is that you can't just keep CVs "on file" without explicit permission, even of former employees you did end up hiring (unless you are preparing a legal case involving claims made in the CV for example).

Data controllers absolutely must delete data by default. The basic rule of GDPR is literally "privacy by default" (or "data minimization"), not "data hoarding by default". There are plenty of exemptions that grant companies the right (or rather require them) to delay the deletion by amounts of time as long as many years (e.g. for tax purposes), but they are only permitted to maintain only the absolute minimum required to fulfill these obligations and they must delete the data as soon as possible once these obligations have been fulfilled.

If you are obligated to keep a set of PII for 10 years, you must delete it in 10 years, unless you (still) have consent to keep it beyond that time limit.

Re: Soft deletion probably isn't worth it

#444

Post by mirages »

jrumbut wrote: post #402
JohnBooty wrote: post #370

Earlier quoted context omitted.

I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Yeah. As far as a user-facing "Undelete" button existing or being used... that's very rare in my experience. What's much more common is a user accidentally deletes some data. They deny they made an error. The developers are blamed. You then have to go on a wild goose chase figuri…

I've long wished for an RDBMS (or perhaps ORM, but I think it would be extra cool at the database level, see below) that does things that don't hyperscale. So many LOB applications (the part of the iceberg under the water) have modestly sized databases with a relatively consistent number of users. The data is small, but often complex, awkwardly structured, highly relational, etc. The challenges these applications fac…

In Oracle you have the flashback tech

  SELECT * FROM employees
  AS OF TIMESTAMP
  TO_TIMESTAMP('2004-04-04 09:30:00', 'YYYY-MM-DD HH:MI:SS')
  WHERE last_name = 'Chung';

Re: Soft deletion probably isn't worth it

#445

Post by sammoorhouse »

jrumbut wrote: post #402
JohnBooty wrote: post #370

Earlier quoted context omitted.

I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Yeah. As far as a user-facing "Undelete" button existing or being used... that's very rare in my experience. What's much more common is a user accidentally deletes some data. They deny they made an error. The developers are blamed. You then have to go on a wild goose chase figuri…

I've long wished for an RDBMS (or perhaps ORM, but I think it would be extra cool at the database level, see below) that does things that don't hyperscale. So many LOB applications (the part of the iceberg under the water) have modestly sized databases with a relatively consistent number of users. The data is small, but often complex, awkwardly structured, highly relational, etc. The challenges these applications fac…

I think what you're looking for is called bitemporality (https://en.wikipedia.org/wiki/Bitemporal_modeling).

It allows you to query a system (some databases support this) in a temporal context. This is useful in financial services where you have market data baked into a price you've offered to a client, where the underlying market data might have been later corrected by a vendor.

So you can query how the world looked last tuesday 4pm; but also - given what we now know, how ought the world have looked last tuesday 4pm.

Interesting stuff.

Re: Soft deletion probably isn't worth it

#446

Post by kgeist »

alerighi wrote: post #216
gmiller123456 wrote: post #172

Earlier quoted context omitted.

The author uses the "no one ever undeleted anything" as the primary justification. I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Either a complaint finally gets around to me as to why the user wanted their account deleted (e.g. feature not working) and it helps to figure out why. Or they're returning and want things set up l…

It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted (or anonymize it in a way that it's no longer possible to associate it back to the particular user). You cannot just keep user information forever "just in case" they are useful again.

>It may be convenient, but under the GDPR is illegal. When an user deletes an account, all the personal data associated with that user must be deleted

There's a grace period of 30 days. We hard-remove after 2 weeks allowing a user to change their mind if they deleted it by accident. More than enough.

Re: Soft deletion probably isn't worth it

#447

Post by closewith »

jandrewrogers wrote: post #389
coder543 wrote: post #365

Earlier quoted context omitted.

Remapping of blocks isn't the problem here. IANAL, but you should be encrypting all the data for GDPR compliance to begin with. With it encrypted, how the storage device chooses to map its blocks is completely irrelevant. When the key goes away, all the data is erased by definition . Throwing away the key is one of the easiest ways to comply with Right to be Forgotten, from what I've seen. "Oh, but the key is still o…

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor performance and scalability. Many classes of major optimization in databases don't work under these constraints. This isn't a novel idea, having been implemented for decades; it has been broadly rejected because it doesn't actually work in real systems without tra…

With respect, your own comment reads as overly dismissive of the regulatory environment introduced by the GDPR. Systems (and companies) have for decades collected and stored personal data with little or no regard for the lifecycle of the data.

Since the introduction of the Regulation, companies must now refrain from the collection of personal data if they cannot satisfy the rights of the data subject, including the right to erasure (which is obviously not an absolute right).

If the technical solutions aren't performant or scalable enough for data controllers, then the result is that they cannot collect personal data. The idea that companies have in the past rejected solutions which protect the rights of data subjects because they were unwilling to accept the technical tradeoffs is exactly why strong regulation such as the GDPR is required.

Frankly, your comment reads as contemptuous to the rights of the end-user. Thankfully, the EU offers protection to its residents from people with that attitude.

Re: Soft deletion probably isn't worth it

#448

Post by dspillett »

michaelt wrote: post #228
dspillett wrote: post #197

Earlier quoted context omitted.

> Since I can no longer rely on ON DELETE CASCADE relationships Cascaded deletes scare me anyway. It only takes one idiot to implement UPSERT as DELETE+INSERT because it seems easier, and child data is lost. You could always use triggers to cascade you soft-delete flags as an alternative method, though that would be less efficient (and more likely to be buggy) than the built-in solution that cascaded deletes are. If…

> It only takes one idiot to implement UPSERT as DELETE+INSERT because it seems easier, and child data is lost. Seems unfortunate to miss out on all the referential integrity benefits of a serious database when hiring standards, training and code reviews should all be preventing idiotic changes. If I’m making a shopping cart system, I want to know every order line belongs to an order, every order belongs to a user an…

> Seems unfortunate to miss out on

You don't lose any referential integrity without cascades. Foreign keys are still enforced, just with an error if an action would break integrity rather than automatic deletes to satisfy the constraint that way.

> when hiring standards, training and code reviews should all be preventing idiotic changes

I was burned by this sort of thing early on, in companies where I had few such luxuries. Even though things are done better now, I'm still paranoid of that one day someone skips procedure and somehow lets a problem through all the QA loops.

> If I’m making a shopping cart system, I want to know every order line belongs to an order, every order belongs to a user and so on.

I take it from the other side: I want to know that if something is referred to elsewhere it can't be deleted until that is resolved.

If a top-level manager leaves I want an error if the hierarchy hasn't been updated before deleting his record¹, rather than his underlings, their underlings, their underling's underlings, … , being deleted when that one person is!

----

[1] Obviously this would normally be a soft-delete, there may be a lot of records referring to such an individual not just other person records. If you actually need to delete them (right to be forgotten etc.) then you need to purge the PII but keep the record so other things still hang together.

Re: Soft deletion probably isn't worth it

#449

Post by cal85 »

JohnBooty wrote: post #151

I've been a software dev since the 90s and at this point, I've learned to basically do things like audit trails and soft deletion by default, unless there's some reason not to. Somebody always wants to undelete something, or examine it to see why it was deleted, or see who changed something, or blah blah blah. It helps the business, it helps you as developer by giving you debug information as well as helping you to c…

> Somebody always wants to undelete something, or examine it to see why it was deleted, or see who changed something, or blah blah blah.

In my experience this happens “rarely”, not “always”.

It can happen, and in some ultra-rare cases the impact of not being able to recover some data might be huge (company-ending, even), and engineers are good at worrying about such edge cases. That’s why we habe protective measures like soft deleting and event sourcing - because of nightmare edge cases, not because we are always having to actually use them. It’s driven by engineers avoiding their worst nightmare: having to say “I’m sorry, I cannot solve this problem for you. The data is gone.” It’s a peace-of-mind thing, not an everyday-need thing.

Re: Soft deletion probably isn't worth it

#450

Post by guiriduro »

jrumbut wrote: post #402
JohnBooty wrote: post #370

Earlier quoted context omitted.

I think this is the part they miss. I've never undeleted a user either, but there have been many times I've gone back to look at something. Yeah. As far as a user-facing "Undelete" button existing or being used... that's very rare in my experience. What's much more common is a user accidentally deletes some data. They deny they made an error. The developers are blamed. You then have to go on a wild goose chase figuri…

I've long wished for an RDBMS (or perhaps ORM, but I think it would be extra cool at the database level, see below) that does things that don't hyperscale. So many LOB applications (the part of the iceberg under the water) have modestly sized databases with a relatively consistent number of users. The data is small, but often complex, awkwardly structured, highly relational, etc. The challenges these applications fac…

For PostgreSQL with WAL logs, or in a service like RDS, can you not do a point in time recovery? Seems like an easy way to go back in time, recover the data you want, dig out any FK referents that might also have been cascade deleted etc. If you also maintained an audit log you should be able to isolate the point where the given data was deleted (if its id is in the audit log), and restore from just before it, to capture the deleted data, query it and reinsert in master.
Post reply on HN