The challenges of soft delete
atlas9.dev
The challenges of soft delete
1–10 of 157 posts
Re: The challenges of soft delete
#2We deal with soft delete in a Mongo app with hundreds of millions of records by simply moving the objects to a separate collection (table) separate from the “not deleted” data.
This works well especially in cases where you don’t want to waste CPU/memory scanning soft deleted records every time you do a lookup.
And avoids situations where app/backend logic forgets to apply the “deleted: false” filter.
Re: The challenges of soft delete
#3We deal with soft delete in a Mongo app with hundreds of millions of records by simply moving the objects to a separate collection (table) separate from the “not deleted” data. This works well especially in cases where you don’t want to waste CPU/memory scanning soft deleted records every time you do a lookup. And avoids situations where app/backend logic forgets to apply the “deleted: false” filter.
I guess that works well with NoSQL. In a relational database it gets harder to move record out if they have relationships with other tables.
Re: The challenges of soft delete
#4The trigger architecture is actually quite interesting, especially because cleanup is relatively cheap. As far as compliance goes, it's also simply to declare that "after 45 days, deletions are permanent" as a catch all, and then you get to keep restores. For example, I think (IANAL), the CCPA gives you a 45 day buffer for right to erasure requests.
Now instead of chasing down different systems and backups, you can simply set ensure your archival process runs regularly and you should be good.
Re: The challenges of soft delete
#5A good solution here (can be) to utilize a view. The underlying table has soft-delete field and the view will hide rows that have been soft deleted. Then the application doesn't need to worry about this concern all over the place.
Re: The challenges of soft delete
#6We deal with soft delete in a Mongo app with hundreds of millions of records by simply moving the objects to a separate collection (table) separate from the “not deleted” data. This works well especially in cases where you don’t want to waste CPU/memory scanning soft deleted records every time you do a lookup. And avoids situations where app/backend logic forgets to apply the “deleted: false” filter.
I guess that works well with NoSQL. In a relational database it gets harder to move record out if they have relationships with other tables.
Eh you could implement this pretty simply with postgres table partitions
Re: The challenges of soft delete
#7Earlier quoted context omitted.
I guess that works well with NoSQL. In a relational database it gets harder to move record out if they have relationships with other tables.
Eh you could implement this pretty simply with postgres table partitions
Ah, that's an interesting idea! I had never considered using partitions. I might write a followup post with these new ideas.
Re: The challenges of soft delete
#8How do you handle schema drift?
The data archive serialized the schema of the deleted object representative the schema in that point in time.
But fast-forward some schema changes, now your system has to migrate the archived objects to the current schema?
Re: The challenges of soft delete
#9A good solution here (can be) to utilize a view. The underlying table has soft-delete field and the view will hide rows that have been soft deleted. Then the application doesn't need to worry about this concern all over the place.
postgres with rls to hide soft deleted records means that most of the app code doesn't need to know or care about them, still issues reads, writes, deletes to the same source table and as far as the app knows its working
Re: The challenges of soft delete
#10Earlier quoted context omitted.
Eh you could implement this pretty simply with postgres table partitions
Ah, that's an interesting idea! I had never considered using partitions. I might write a followup post with these new ideas.
There are a bunch of caveats around primary keys and uniqueness but I suspect it could be made to work depending on your data model.