Related from a few years ago: PAGNIs, for Probably Are Gonna Need Its - https://simonwillison.net/2021/Jul/1/pagnis/
YAGRI: You are gonna read it
81–90 of 161 posts
Re: YAGRI: You are gonna read it
#82OT As the great Alone ("Last Psychiatrist") said, "if you read it, it's for you" - IYRIIFY
Re: YAGRI: You are gonna read it
#83These are not decisions that should be taken solely by whoever is programming the backend. They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong. If the product owner wants to start tracking every change and by who, that can completely change your database requirements. So have that conversa…
> don't do it as some half-built just-in-case "favor" to a future programmer who may very well have to rip it out.
in other words - YAGNI !
Re: YAGRI: You are gonna read it
#84While I like the YAGRI principle very much, I find that adding - updated_at - deleted_at (soft deletes) - created_by etc - permission used during CRUD to every table is a solution weaker than having a separate audit log table. I feel that mixing audit fields with transactional data in the same table is a violation of the separation of concerns principle. In the proposed solution, updated_at only captures the last cha…
Event sourcing also works great. You don't need an audit log per se if you already track a history of all commands that introduced changes to your system.
Yep. But Event Sourcing comes with its own set of other problems.
Also, I don't think this would apply to OP's post: with Event Sourcing you would not even have those DB tables.
Re: YAGRI: You are gonna read it
#85While I like the YAGRI principle very much, I find that adding - updated_at - deleted_at (soft deletes) - created_by etc - permission used during CRUD to every table is a solution weaker than having a separate audit log table. I feel that mixing audit fields with transactional data in the same table is a violation of the separation of concerns principle. In the proposed solution, updated_at only captures the last cha…
An audit log table often takes a huge amount of space compared to simple fields on the records so there are tradeoffs. Which solution is best depends on how important change logs are.
Re: YAGRI: You are gonna read it
#86These are not decisions that should be taken solely by whoever is programming the backend. They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong. If the product owner wants to start tracking every change and by who, that can completely change your database requirements. So have that conversa…
Some things are trivial and nearly free - created_at, updated_at. I don't think engineers need to bring trivialities like this to a "product owner". Own your craft.
A soft delete might not be, for compliance reasons (GDPR and the like). Otherwise I agree.
Re: YAGRI: You are gonna read it
#87*_at and *_by fields in SQL are just denormalization + pruning patterns consolidated, right? Do the long walk: Make the schema fully auditable (one record per edit) and the tables normalized (it will feel weird). Then suffer with it, discover that normalization leads to performance decrease. Then discover that pruned auditing records is a good middle ground. Just the last edit and by whom is often enough (ominous for…
My current state is have the database be the current state and use logical replication (CDC) to keep the log of changes in case you need it
The problem with this is the audit log is only at the CRUD level which is often too low. Ambiguities can arise. For example if the question is "who published the article" do you look for a create or do you look for an update with published=true? It's even worse when you consider the schema can change over time, so both can be correct but at different points in time. Event sourcing is the way if you want to capture business-level events.
Re: YAGRI: You are gonna read it
#88I don't like general advice like this, because it's too general. For many, it's probably good advice. For others, not so much.
Anyone who has worked at a small company selling to large B2B SaaS can attest we get like 20 hits a day on a busy day. Most of that is done by one person in one company, who is probably also the only person from said company you've ever talked to.
From that lens, this is all overkill. It's not bad advice, it's just that it will get quoted for scenarios it doesn't apply. Which also apply to K8S, or microservices at large even, and most 'do as I say' tech blogs.
Re: YAGRI: You are gonna read it
#89[dead]
Re: YAGRI: You are gonna read it
#90*_at and *_by fields in SQL are just denormalization + pruning patterns consolidated, right? Do the long walk: Make the schema fully auditable (one record per edit) and the tables normalized (it will feel weird). Then suffer with it, discover that normalization leads to performance decrease. Then discover that pruned auditing records is a good middle ground. Just the last edit and by whom is often enough (ominous for…
Another option is audit info could go to another table or datastore entirely. If you never use it, that data can be dumped to s3 glacier periodically (e.g. after 90 days). By losing the foreign key you gain flexibility in what you audit. Maybe audit the operation and not the 20 writes it causes.
Fair enough, but now your application is relying on 100% uptime of AWS and S3 and no network failures in between. And what happens if your transaction goes through, but the request to AWS doesn’t? What happens if another operation mutates the target meanwhile before you can retry with current state? Your app is also slowing down since it needs to send the events to S3 and guarantee they got there. Now you are reinventing two-stage commits. Unless you aren’t actually making an audit log and don’t care if events are guaranteed to be logged?
So like OP said, no silver bullets exist for auditing.