Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YAGRI: You are gonna read it

scottantipa.com

Post reply on HN
Print view View original

61–70 of 161 posts

Re: YAGRI: You are gonna read it

#61

Post by lud_lite »

alganet wrote: post #48

*_at and *_by fields in SQL are just denormalization + pruning patterns consolidated, right? Do the long walk: Make the schema fully auditable (one record per edit) and the tables normalized (it will feel weird). Then suffer with it, discover that normalization leads to performance decrease. Then discover that pruned auditing records is a good middle ground. Just the last edit and by whom is often enough (ominous for…

Another option is audit info could go to another table or datastore entirely.

If you never use it, that data can be dumped to s3 glacier periodically (e.g. after 90 days).

By losing the foreign key you gain flexibility in what you audit. Maybe audit the operation and not the 20 writes it causes.

Re: YAGRI: You are gonna read it

#62

Post by lud_lite »

jfengel wrote: post #41
reillyse wrote: post #4

One thing I do quite frequently which is related to this (and possibly is a pattern in rails) is to use times in place of Booleans. So is_deleted would contain a timestamp to represent the deleted_at time for example. This means you can store more information for a small marginal cost. It helps that rails will automatically let you use it as a Boolean and will interpret a timestamp as true.

I consider booleans a code smell. It's not a bug, but it's a suggestion that I'm considering something wrong. I will probably want to replace it with something more meaningful in the future. It might be an enum, a subclass, a timestamp, refactoring, or millions of other things, but the Boolean was probably the wrong thing to do even if I don't know it yet.

Booleans also force the true/false framing.

E.g. a field called userCannotLoginWithoutOTP.

Then in code "if not userCannotLoginWithoutOTP or otpPresent then..."

Thus may seem easy until you have a few flags to combine and check.

An enum called LoginRequirements with values Password, PasswordAndOTP is one less negation and easier to read.

Re: YAGRI: You are gonna read it

#63

Post by awesome_dude »

alganet wrote: post #48

*_at and *_by fields in SQL are just denormalization + pruning patterns consolidated, right? Do the long walk: Make the schema fully auditable (one record per edit) and the tables normalized (it will feel weird). Then suffer with it, discover that normalization leads to performance decrease. Then discover that pruned auditing records is a good middle ground. Just the last edit and by whom is often enough (ominous for…

But wait, there's Event Driven Architectures and Event Sourcing, meaning that the events are your log of edits!

Re: YAGRI: You are gonna read it

#64

Post by klysm »

datadrivenangel wrote: post #54
klysm wrote: post #50

Earlier quoted context omitted.

My current state is have the database be the current state and use logical replication (CDC) to keep the log of changes in case you need it

It is interesting thinking about record changes as a spectrum towards application logs. At some point too much detail is expensive to store, and you must adopt an archival strategy.

Really depends on the app. If you have a low throughput line-of-business kind of application you can probably get away with storing everything.

Re: YAGRI: You are gonna read it

#65

Post by klysm »

alganet wrote: post #56
klysm wrote: post #50

Earlier quoted context omitted.

My current state is have the database be the current state and use logical replication (CDC) to keep the log of changes in case you need it

If you see it from the pure SQL point of view, you are in the "blame database engines and adopt an experimental solution". It is the point where you give up modeling the audit as part of the systems tables. The drawbacks of this choice are often related to retrieval. It depends on the engine. I once maintained a system that kept a fully working log replicated instance delayed by 24h, ready for retrieval queries, in a…

Yeah 100% giving up on pure SQL to solve the problem, mainly from the perspective that doing full versioning etc. in SQL is really damn hard.

Re: YAGRI: You are gonna read it

#66

Post by alganet »

awesome_dude wrote: post #63
alganet wrote: post #48

*_at and *_by fields in SQL are just denormalization + pruning patterns consolidated, right? Do the long walk: Make the schema fully auditable (one record per edit) and the tables normalized (it will feel weird). Then suffer with it, discover that normalization leads to performance decrease. Then discover that pruned auditing records is a good middle ground. Just the last edit and by whom is often enough (ominous for…

But wait, there's Event Driven Architectures and Event Sourcing, meaning that the events are your log of edits!

Doesn't that also falls on the "blame the database engines and go for an experimental solution"?

I'm not saying databases are blameless. It's just that experiencing the issues they have by yourself is rewarding!

There is also a walk before the long walk of databases. Store things in text files and use basic tools (cat, sed, sh...).

The event driven stuff (like Kafka) reminds me of that. I am not very familiar with it though, just played a little bit with it once or twice.

Re: YAGRI: You are gonna read it

#67

Post by willismichael »

000ooo000 wrote: post #49
im_anon_on_hn wrote: post #39

As an acronym, it's easy to be misremembered as "You ARENT gonna read it" (based on the popularity of yagni) - and have the opposite advice spread..

How about: IIPTWNIRTIoSoDIQMBSNRTL (It Is Probable That While Not Immediately Required The Implementation of Storage of Data In Question May Be Simpler Now Rather Than Later) I've gone ahead and included additional detail in the acronym in the event that the clarity is required later, as this would be difficult to retrofit into a shorter, more-established acronym.

Why do I feel like that should be a cheat code in a FPS game?

Re: YAGRI: You are gonna read it

#68

Post by pmontra »

swagasaurus-rex wrote: post #11
evanmoran wrote: post #10

Just curious, how do people feel about this general style of soft deletes currently? Do people still use these in production or prefer to just delete fully or alternatively move deleted rows to a separate tables / schema? I find the complexity to still feel awkward enough that makes me wonder if deleted_at is worth it. Maybe there are better patterns out there to make this cleaner like triggers to prevent deletion, s…

I think soft deletes using timestamptz are a good thing. Deleting rows directly could mean you're breaking references. For example, say you have a product that the seller wants to delete. Well, what happens if customers have purchased that product? You still want it in the database, and you still want to fulfill the orders placed. Your backend can selectively query for products, filter out deleted_at for any customer…

> what happens if customers have purchased that product? You still want it in the database, and you still want to fulfill the orders placed.

You might still want to show to those customers their purchase history including what they bought 25 years ago. For example, my ISP doesn't have anymore that 10 Mb/s fiber optic product I bought im 2000, because it was superseded by 100 Mb/s products and then by 1 Gb/s ones. It's also not my ISP anymore but I use it for the SIM in my phone. That also accumulated a number of product changes along the years.

And think about the inventory of eshops with a zillion products and the archive of the pady orders. Maybe they keep the last few years, maybe everything until the db gets too large.

Re: YAGRI: You are gonna read it

#69

Post by lexh »

dml2135 wrote: post #24
zie wrote: post #18

Earlier quoted context omitted.

If you have a good audit log, it really doesn't matter. You can always restore it if need be. If you have no audit log(or a bad one), like lots of apps, then you have to care a lot. Personally, I just implement a good audit log and then I just delete with impunity. Worst case scenario, someone(maybe even me) made a mistake and I have to run undo_log_audit() with the id of the audit log entry I want to put back. Nearl…

Can you share more about what makes a good audit log? My company doesn’t currently have one and I’m a little lost on where to start. Should this be at the application code level, or the ORM, or the database itself?

It is Postgres specific, but I’ve gotten a lot of mileage out of the advice in this article:

https://supabase.com/blog/postgres-audit

Re: YAGRI: You are gonna read it

#70

Post by awesome_dude »

alganet wrote: post #66
awesome_dude wrote: post #63

Earlier quoted context omitted.

But wait, there's Event Driven Architectures and Event Sourcing, meaning that the events are your log of edits!

Doesn't that also falls on the "blame the database engines and go for an experimental solution"? I'm not saying databases are blameless. It's just that experiencing the issues they have by yourself is rewarding! There is also a walk before the long walk of databases. Store things in text files and use basic tools (cat, sed, sh...). The event driven stuff (like Kafka) reminds me of that. I am not very familiar with it…

Kind of, the WAL in postgres is effectively an event log, and many people keep replicas of it for backup reasons, which is auditable, kind of meaning that an EDA/Event source is just a shinier version of that?
Post reply on HN