Live data from Hacker News

The challenges of soft delete

atlas9.dev

111–120 of 157 posts

Re: The challenges of soft delete

#111

This might stem from the domain I work in (banking), but I have the opposite take. Soft delete pros to me: * It's obvious from the schema: If there's a `deleted_at` column, I know how to query the table correctly (vs thinking rows aren't DELETEd, or knowing where to look in another table) * One way to do things: Analytics queries, admin pages, it all can look at the same set of data, vs having separate handling for h…

The core system at my previous employer (an insurance company) worked along the lines of the solution you outline at the end: each table is an append only log of point in time information about some object. So the current state is in the row with the highest timestamp, and all previous stars can be observed with appropriate filters. It’s a really powerful approach.

So basically something like this?

(timestamp, accountNumber, value, state)

And then you just

SELECT state FROM Table WHERE accountNumber = ... ORDER BY timestamp DESC LIMIT 1

right?

Re: The challenges of soft delete

#112
post #43

This might stem from the domain I work in (banking), but I have the opposite take. Soft delete pros to me: * It's obvious from the schema: If there's a `deleted_at` column, I know how to query the table correctly (vs thinking rows aren't DELETEd, or knowing where to look in another table) * One way to do things: Analytics queries, admin pages, it all can look at the same set of data, vs having separate handling for h…

If you're implementing immutable DB semantics maybe you should consider Datomic or alternatives because then you get that for free, for everything, and you also get time travel which is an amazing feature on top. It lets you be able to see the full, coherent state of the DB at any moment!

My understanding is that Datomic uses something like Postgres as a storage backend. Am I right?

Also, it doesn't support non-immutable use cases AFAIK, so if you need both you have to use two database technologies (interfaces?), which can add complexity.

Re: The challenges of soft delete

#113

Earlier quoted context omitted.

The core system at my previous employer (an insurance company) worked along the lines of the solution you outline at the end: each table is an append only log of point in time information about some object. So the current state is in the row with the highest timestamp, and all previous stars can be observed with appropriate filters. It’s a really powerful approach.

So basically something like this? (timestamp, accountNumber, value, state) And then you just SELECT state FROM Table WHERE accountNumber = ... ORDER BY timestamp DESC LIMIT 1 right?

Yeah, basically. The full system actually has more date stuff going on, to support some other more advanced stuff than just tracking objects themselves, but that's the overall idea. When you need to join stuff it can be annoying to get the SQL right in order to join the correct records from a different table onto your table of interest (thank Bob for JOIN LATERAL), but once you get the hang of it it's fairly straightforward. And it gives you the full history, which is great.

Re: The challenges of soft delete

#114

Earlier quoted context omitted.

So basically something like this? (timestamp, accountNumber, value, state) And then you just SELECT state FROM Table WHERE accountNumber = ... ORDER BY timestamp DESC LIMIT 1 right?

Yeah, basically. The full system actually has more date stuff going on, to support some other more advanced stuff than just tracking objects themselves, but that's the overall idea. When you need to join stuff it can be annoying to get the SQL right in order to join the correct records from a different table onto your table of interest (thank Bob for JOIN LATERAL), but once you get the hang of it it's fairly straight…

Sounds cool! Do you keep all data forever in the same table? I assume you need long retention, so do you keep everything in the same table for years or do you keep a master table for, let's say, the current year and then "rotate" (like logrotate) previous stuff to other tables?

Even with indices, a table with, let's say, a billion rows can be annoying to traverse.

Re: The challenges of soft delete

#116
There are tables at $dayjob with both (begin, end) and also (incept, expire) fields. It's "on such-and-such date, X was true", but also allows for "as-of Z date, we believed that...".

Also you can have most data being currently unused even without being flagged deleted. Like if I go in to our ticketing system, I can still see my old requests that were closed ages ago.

Re: The challenges of soft delete

#117

[flagged]

I'm struggling to see your point. CREATE VIEW not only helps, yes, indeed it's oftentimes exactly all you need. If you have multiple access patterns, like having to "actually query deleted records" sometimes, somewhere, at some point, someone would have to maintain invariants on these access patterns. This is not rocket science. The heart of the matter is that SWE's cannot handle schema/basic SQL to save their lives, whilst analysts/BI guys/whomever actually somewhat well-versed in SQL, have very little grasp on the inner working of a database, and carry with themselves idiosyncrasies coming all the way back from the 90's.

The pot is calling the kettle black.

Forget about soft deletes for a hot minute. I can give you another super basic example where in my experience SWE's and BI guys both lose the plot: Type 2 slowly-changing dimensions. This is actually heavily related to soft deletes, and much more common as far as access patterns are concerned. Say, you want to support data updates without losing information unless specified by a retention policy. For argument's sake, let's say you want to keep track of edits in the user profile. How do you do it? If you go read up on Stackoverflow, or whatever, you will come across the idea that did more violence to schemas worldwide than anything else in existence, "audit table." So instead of performing a cheap INSERT on a normalised data structure every time you need to make a change, and perhaps reading up-to-date data from a view, you're now performing costly UPDATE, and additional INSERT anyway. Why? Because apparently DISTINCT ON and composite primary keys are black magic (and anathema to ORM's in general.) If you think on BI side they're doing any better, you think wrong! To them, DISTINCT ON is oftentimes a mystery no less. One moment, blink, there you go, back in the subquery hell they call home.

Databases are beautiful, man.

It's a shame they are not treated with more respect that they deserve.

Re: The challenges of soft delete

#118

Earlier quoted context omitted.

Yeah, basically. The full system actually has more date stuff going on, to support some other more advanced stuff than just tracking objects themselves, but that's the overall idea. When you need to join stuff it can be annoying to get the SQL right in order to join the correct records from a different table onto your table of interest (thank Bob for JOIN LATERAL), but once you get the hang of it it's fairly straight…

Sounds cool! Do you keep all data forever in the same table? I assume you need long retention, so do you keep everything in the same table for years or do you keep a master table for, let's say, the current year and then "rotate" (like logrotate) previous stuff to other tables? Even with indices, a table with, let's say, a billion rows can be annoying to traverse.

I wasn’t involved in the day to day operations of the system, but it had records going back to the 90s at least I think. I think data related to non accepted offers were deleted fairly quickly (since they didn’t end up being actual customers), but outside of that I think everything was kept more or less indefinitely.

Re: The challenges of soft delete

#119
post #117

[flagged]

I'm struggling to see your point. CREATE VIEW not only helps, yes, indeed it's oftentimes exactly all you need. If you have multiple access patterns, like having to "actually query deleted records" sometimes, somewhere, at some point, someone would have to maintain invariants on these access patterns. This is not rocket science. The heart of the matter is that SWE's cannot handle schema/basic SQL to save their lives,…

I believe this all stems from primordial SQL focusing on storage efficiency, and now it’s kinda hard to retrofit better data modeling ideas without better affordances.

If I started from scratch, I would get rid of UPDATE and DELETE (these would be only very special cases for data privacy), and instead focus on first class views (either batch copy or streaming) and retention policies.

Re: The challenges of soft delete

#120
Could Postgres provide a mechanism where delete works as you'd expect but you can add WITH DELETED keyword to a SELECT and it returns everything even deleted records? I guess migrations are still an issue if you want to change the structure of the DB but maybe you could provide these as part of the database too - so INSERT INTO table(col1, col2, newCol...) FROM DELETED (col1, col2, newDataNotInDeleted) WHERE id = 123 CASCADE; or something like this.

There should be a preferred way to handle this as these are clearly real issues that the database should help you to deal with.

Post reply on HN