Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YAGRI: You are gonna read it

scottantipa.com

Post reply on HN
Print view View original

51–60 of 161 posts

Re: YAGRI: You are gonna read it

#51

Post by 000ooo000 »

code_biologist wrote: post #22
mkhalil wrote: post #7

Event-sourcing solves this. And with how cheap storage is, it should be more prevalent in the industry. IMO the biggest thing holding it back is that there isn't a framework that's plug-and-play (say like Next.js is to React) that provides people with that ability. I've been working on one in Typescript (with eventual re-writes in other langs. like Rust and Go), but it's difficult even coming up with conventions.

Event sourcing is an expensive solution and I don't mean from a storage perspective — it burns engineering cognitive horsepower quickly on things that don't matter. Do it if you're in finance or whatever. Having been burned by my own "let's event source" impulse on data change tracking systems, I now prefer less sophisticated solutions. Figuring out how to deal with slow projections, watching a projection rebuild go…

Appreciate your perspective, and it makes me wish there was some kind of online 'engineers learning from their mistakes' forum (rare to see "I burned myself"). To hear hard won knowledge distilled like this is a nice reminder to spend ones complexity budget wisely.

Re: YAGRI: You are gonna read it

#52

Post by __float »

tcdent wrote: post #20
SOLAR_FIELDS wrote: post #15

Earlier quoted context omitted.

“Reliable migrations” almost seems like an oxymoron. Migrations are complicated, difficult and error prone. I think there’s a good takeaway here around good initial schema design practices. The less you have to morph your schema overtime, the less of those risky migrations need to run.

My experience over the last decade has been different. Use a popular framework. Run it against your test database. Always keep backups in case something unforseen happens. Something especially trivial like adding additional columns is a solved problem.

My experience has not been so smooth. Migrations are reasonable, but they're not free and "always keeps backups" sounds like you'd tolerate downtime more than I would.

Even in the best case (e.g. basic column addition), the migration itself can be "noisy neighbors" for other queries. It can cause pressure on downstream systems consuming CDC (and maybe some of those run queries too, and now your load is even higher).

Re: YAGRI: You are gonna read it

#54

Post by datadrivenangel »

klysm wrote: post #50
alganet wrote: post #48

*_at and *_by fields in SQL are just denormalization + pruning patterns consolidated, right? Do the long walk: Make the schema fully auditable (one record per edit) and the tables normalized (it will feel weird). Then suffer with it, discover that normalization leads to performance decrease. Then discover that pruned auditing records is a good middle ground. Just the last edit and by whom is often enough (ominous for…

My current state is have the database be the current state and use logical replication (CDC) to keep the log of changes in case you need it

It is interesting thinking about record changes as a spectrum towards application logs. At some point too much detail is expensive to store, and you must adopt an archival strategy.

Re: YAGRI: You are gonna read it

#55

Post by jchw »

noman-land wrote: post #40
jchw wrote: post #33

Earlier quoted context omitted.

32-bit UNIX timestamps are often signed so you can actually go before that, but most UNIX timestamps are 64-bit now, which can represent quite a larger range. And SQL datetime types might have a totally different range. Not that it really matters; deleted_at times for your database records will rarely predate the existence of said database.

It's not about the scale, it's that `if (0)` will evaluate to `false` in many languages.

In addition to the sibling comment, which is exactly right (you should be using a nullable column here, if you're using SQL, for multiple reasons) I reckon this a design issue in the programming language that is largely unrelated to how you model the database. It's pretty easy to run into bugs especially if you compound it with other quirky APIs, like strcmp: `if (strcmp(a, b)) // forgot to do == 0; accidentally backwards!` -- So really, you just don't have much of a choice other than to tread carefully and enable compiler warnings. Personally in this case I'd use an Optional wrapper around the underlying timestamp type anyways, if I needed to be able to represent the UNIX timestamp at 0 as well as an empty state.

Re: YAGRI: You are gonna read it

#56

Post by alganet »

klysm wrote: post #50
alganet wrote: post #48

*_at and *_by fields in SQL are just denormalization + pruning patterns consolidated, right? Do the long walk: Make the schema fully auditable (one record per edit) and the tables normalized (it will feel weird). Then suffer with it, discover that normalization leads to performance decrease. Then discover that pruned auditing records is a good middle ground. Just the last edit and by whom is often enough (ominous for…

My current state is have the database be the current state and use logical replication (CDC) to keep the log of changes in case you need it

If you see it from the pure SQL point of view, you are in the "blame database engines and adopt an experimental solution".

It is the point where you give up modeling the audit as part of the systems tables.

The drawbacks of this choice are often related to retrieval. It depends on the engine.

I once maintained a system that kept a fully working log replicated instance delayed by 24h, ready for retrieval queries, in addition to regular disk backups (slow costy retrieval).

I am more developer than DBA, so I can probably speak more about modeling solutions than infra-centric solutions.

Re: YAGRI: You are gonna read it

#57

Post by Hasnep »

shaunofthedev wrote: post #26

The perils of UI design wagging the horse. I like the heuristics described here. However if these things aren't making it into a product spec where appropriate, then I smell some dysfunction that goes beyond what's being stored by default. Product need (expressed as spec, design, etc) should highlight the failure cases where we would expect fields like these to be surfaced. I'd hope that any given buisness shouldn't…

"Wagging the horse" is a great turn of phrase, better than "putting the cart before the dog."

Re: YAGRI: You are gonna read it

#58

Post by crdrost »

xp84 wrote: post #44
jfengel wrote: post #41

Earlier quoted context omitted.

I consider booleans a code smell. It's not a bug, but it's a suggestion that I'm considering something wrong. I will probably want to replace it with something more meaningful in the future. It might be an enum, a subclass, a timestamp, refactoring, or millions of other things, but the Boolean was probably the wrong thing to do even if I don't know it yet.

This seems at first like a controversial idea, but the more I think about it the more I like this thought technology. Merely the idea of asking myself if there's a better way to store a fact like that will potentially improve designs.

The enum idea is often wise; also: for just an example that has probably occurred a hundred thousand times across the world in various businesses...

Original design: store a row that needs to be reported to someone, with an is_reported column that is boolean.

Problem: one day for whatever reason the ReporterService turns out to need to run two of these in parallel. Maybe it's that the reporting is the last step after ingestion in a single service and we need to ingest in parallel. Maybe it's that there are too many reports to different people and the reports themselves are parallelizable (grab 5 clients, grab unreported rows that foreign key to them, report those rows... whoops sometimes two processes choose the same client!)... Maybe it's just that these are run in Kubernetes and if the report happens when you're rolling pods then the request gets retried by both the dying pod and the new pod.

Alternative to boolean: unreported and reported records both live in the `foo` table and then a trigger puts a row for any new Foos into the `foo_unreported` table. This table can now store a lock timestamp, a locker UUID, and denormalize any columns you need (client_id) to select them. The reporter UPDATEs a bunch of rows reserving them, SELECTs whatever it has successfully reserved, reports them, then DELETEs them. It reserves rows where the lock timestamp IS NULL or is less than now minus 5 minutes, and the Reporter itself runs with a 5 minute timeout. The DB will do the barest amount of locking to make sure that two UPDATES don't conflict, there is no risk of deadlock, and the Boolean has turned into whether something exists in a set or not.

A similar trick is used in the classic Python talk “Stop Writing Classes” by @jackdied where a version of The Game of Life is optimized by saying that instead of holding a big 2D array of true/false booleans on a finite gameboard, we'll hold an infinite gameboard with a set of (x,y) pairs of living cells which will internally be backed by a hashmap.

Re: YAGRI: You are gonna read it

#59

Post by guy4261 »

Author is very kind! In practice, many times I saw only the CR/CRU of CRUD getting implemented.

For example: as a company aspires to launch its product, one of the first features implemented in any system is to add a new user. But when the day comes when a customer leaves, suddenly you discover no one implemented off-boarding and cleanup of any sort.

Re: YAGRI: You are gonna read it

#60

Post by crazygringo »

These are not decisions that should be taken solely by whoever is programming the backend.

They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong.

If the product owner wants to start tracking every change and by who, that can completely change your database requirements.

So have that conversation properly. Then decide it's either not worth it and don't add any of these "extra" fields you "might" need, or decide it is and fully spec it out and how much additional time and effort it will be to do it as a proper feature. But don't do it as some half-built just-in-case "favor" to a future programmer who may very well have to rip it out.

On a personal project, do whatever you want. But on something professional, this stuff needs to be specced out and accounted for. This isn't a programming decision, it's a product decision.

Post reply on HN