Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YAGRI: You are gonna read it

scottantipa.com

Post reply on HN
Print view View original

101–110 of 161 posts

Re: YAGRI: You are gonna read it

#101

Post by PeterStuer »

evanmoran wrote: post #10

Just curious, how do people feel about this general style of soft deletes currently? Do people still use these in production or prefer to just delete fully or alternatively move deleted rows to a separate tables / schema? I find the complexity to still feel awkward enough that makes me wonder if deleted_at is worth it. Maybe there are better patterns out there to make this cleaner like triggers to prevent deletion, s…

There can be legal requirements to retain data for a specified time for law enforcement and audits, while at the same time other legal requirements have you requiring to delete data upon customer request.

Doing this with pure 'hard' deletes is not possible, unless you maintain 2 different tables, one of which would still have the soft delete explicit or implicit. You could argue the full db log would contain the data for the former requirement, but while academicly correct this does not fly in practice.

Re: YAGRI: You are gonna read it

#102

Post by Cthulhu_ »

crazygringo wrote: post #60

These are not decisions that should be taken solely by whoever is programming the backend. They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong. If the product owner wants to start tracking every change and by who, that can completely change your database requirements. So have that conversa…

A product owner may (probably does not) have these things on their radar, it's up to a developer to inform them of industry best practices like these.

Re: YAGRI: You are gonna read it

#103

Post by lud_lite »

yes_man wrote: post #90
lud_lite wrote: post #61

Earlier quoted context omitted.

Another option is audit info could go to another table or datastore entirely. If you never use it, that data can be dumped to s3 glacier periodically (e.g. after 90 days). By losing the foreign key you gain flexibility in what you audit. Maybe audit the operation and not the 20 writes it causes.

Fair enough, but now your application is relying on 100% uptime of AWS and S3 and no network failures in between. And what happens if your transaction goes through, but the request to AWS doesn’t? What happens if another operation mutates the target meanwhile before you can retry with current state? Your app is also slowing down since it needs to send the events to S3 and guarantee they got there. Now you are reinven…

Correct. This is a system design problem. You want this to be transactional and work at scale? That might be hard to achieve. Maybe if the data can be partioned then each node handles its own auditing in a table ad part of the transaction. There are many possibilities. Allowing inconsistently might be OK too depending on what is required.

Re: YAGRI: You are gonna read it

#104

Post by wodenokoto »

arialdomartini wrote: post #77

While I like the YAGRI principle very much, I find that adding - updated_at - deleted_at (soft deletes) - created_by etc - permission used during CRUD to every table is a solution weaker than having a separate audit log table. I feel that mixing audit fields with transactional data in the same table is a violation of the separation of concerns principle. In the proposed solution, updated_at only captures the last cha…

I kinda agree, but don’t underestimate the power of having things where people are looking.

Put your documentation in doc strings where the function is defined - don’t have a separate file in a separate folder for that. It might separate concerns, but no one is looking there.

Similarly if those fields aren’t nullable, someone trying to add new rows will have to fill in something for those metadata fields - and that something will now very likely be what’s needed, rather than not pushing anything to the audit table.

Obviously your app can outgrow these simple columns, but you’re getting value now.

Re: YAGRI: You are gonna read it

#105

Post by motorest »

nine_k wrote: post #93
motorest wrote: post #79

Earlier quoted context omitted.

Event sourcing also works great. You don't need an audit log per se if you already track a history of all commands that introduced changes to your system.

Event sourcing and "the right to be forgotten" are not always easy to marry.

> Event sourcing and "the right to be forgotten" are not always easy to marry.

The absolute basics is to support snapshots and event replay. This is hardly rocket science.

Re: YAGRI: You are gonna read it

#106

Post by danielPort9 »

hamandcheese wrote: post #71
crazygringo wrote: post #60

These are not decisions that should be taken solely by whoever is programming the backend. They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong. If the product owner wants to start tracking every change and by who, that can completely change your database requirements. So have that conversa…

Some things are trivial and nearly free - created_at, updated_at. I don't think engineers need to bring trivialities like this to a "product owner". Own your craft.

I think the tricky part lies on knowing which things can be done without consulting any product owner. I agree that created_at and updated_at don’t cause any harm. deleted_at on the other hand cannot be decided by engineers only (mainly because of GDPR reasons: if something is expected to be totally deleted, then that must be it). As usual, these kind of things are obvious to engineers with years of experience , not so much to newcomers.

Re: YAGRI: You are gonna read it

#107

Post by IanCal »

hamandcheese wrote: post #71
crazygringo wrote: post #60

These are not decisions that should be taken solely by whoever is programming the backend. They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong. If the product owner wants to start tracking every change and by who, that can completely change your database requirements. So have that conversa…

Some things are trivial and nearly free - created_at, updated_at. I don't think engineers need to bring trivialities like this to a "product owner". Own your craft.

Although those can be more complicated, and it should be clear what they're for and why they exist. Will this result in an object having an updated_at timestamp elsewhere in a larger application? Is it clear which properties that refers to?

Re: YAGRI: You are gonna read it

#108

Post by motorest »

arialdomartini wrote: post #84
motorest wrote: post #79

Earlier quoted context omitted.

Event sourcing also works great. You don't need an audit log per se if you already track a history of all commands that introduced changes to your system.

Yep. But Event Sourcing comes with its own set of other problems. Also, I don't think this would apply to OP's post: with Event Sourcing you would not even have those DB tables.

The DB tables suggested by OP are a kin to snapshots, whereas each event would require a separate data store. OP is trying to shoehorn event history into the snapshots, which hardly makes any sense.

Re: YAGRI: You are gonna read it

#109

Post by KronisLV »

I don't get why all of the big RDBMSes (PostgreSQL, MariaDB/MySQL, SQL Server, Oracle, ...) don't seem to have built in support for soft deletes up front and center?

  CREATE TABLE ... WITH SOFT DELETES
Where the regular DELETE wouldn't get rid of the data for real but rather you could query the deleted records as well, probably have timestamps for everything as a built in low level feature, vs having to handle this with a bunch of ORMs and having to remember to put AND deleted_at IS NULL in all of your custom views.

If we like to talk about in-database processing so much, why don't we just put the actual common features in the DB, so that toggling them on or off doesn't take a bunch of code changes in app, or that you'd even be able to add soft deletes to any legacy app that knows nothing of the concept, on a per table basis or whatever.

Re: YAGRI: You are gonna read it

#110

Post by hombre_fatal »

hamandcheese wrote: post #71
crazygringo wrote: post #60

These are not decisions that should be taken solely by whoever is programming the backend. They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong. If the product owner wants to start tracking every change and by who, that can completely change your database requirements. So have that conversa…

Some things are trivial and nearly free - created_at, updated_at. I don't think engineers need to bring trivialities like this to a "product owner". Own your craft.

I never worked at a place with product owners, but their post made me appreciate my roles where I'm trusted to help design the product myself. Yeesh.

Being unable to even call the shot of whether a database table should have an updated_at or soft-delete sounds like a Dilbertian hellscape to me.

Post reply on HN