Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YAGRI: You are gonna read it

scottantipa.com

Post reply on HN
Print view View original

151–160 of 161 posts

Re: YAGRI: You are gonna read it

#151

Post by SOLAR_FIELDS »

__float wrote: post #52
tcdent wrote: post #20

Earlier quoted context omitted.

My experience over the last decade has been different. Use a popular framework. Run it against your test database. Always keep backups in case something unforseen happens. Something especially trivial like adding additional columns is a solved problem.

My experience has not been so smooth. Migrations are reasonable, but they're not free and "always keeps backups" sounds like you'd tolerate downtime more than I would. Even in the best case (e.g. basic column addition), the migration itself can be "noisy neighbors" for other queries. It can cause pressure on downstream systems consuming CDC (and maybe some of those run queries too, and now your load is even higher).

Anything with state is going to be hard to get right. Couple sticky schema changes to that state and you’re looking at a lot of potential ways of things can go wrong. Downs being unnecessarily destructive, rollback corrupts data, migrations applied in wrong order. Everyone tangential to working with any sort of migrations system has a war story (or a few) of the creative way that the state got wrecked.

Here’s one of mine: Postgres change applied fine in unit and integration and dev but not prod because the shape of the data (enum) did not conform to the new constraint.

Another would be a monorepo that had 5-6 services that talk across db’s to each other caused dev to apply the wrong migration to the wrong HEAD, mixing up the db’s. That was a fun one to sort out

Re: YAGRI: You are gonna read it

#152

Post by johnthescott »

KronisLV wrote: post #109

I don't get why all of the big RDBMSes (PostgreSQL, MariaDB/MySQL, SQL Server, Oracle, ...) don't seem to have built in support for soft deletes up front and center? CREATE TABLE ... WITH SOFT DELETES Where the regular DELETE wouldn't get rid of the data for real but rather you could query the deleted records as well, probably have timestamps for everything as a built in low level feature, vs having to handle this wi…

old problem. in postgresql many ways already exist to keep deleted data hanging around, not the least of which is explicity archiving the delete transaction in an archive table. for legacy code that can not be changed: triggers, row level security, tailing logical replication log.

Re: YAGRI: You are gonna read it

#153

Post by SOLAR_FIELDS »

tcdent wrote: post #20
SOLAR_FIELDS wrote: post #15

Earlier quoted context omitted.

“Reliable migrations” almost seems like an oxymoron. Migrations are complicated, difficult and error prone. I think there’s a good takeaway here around good initial schema design practices. The less you have to morph your schema overtime, the less of those risky migrations need to run.

My experience over the last decade has been different. Use a popular framework. Run it against your test database. Always keep backups in case something unforseen happens. Something especially trivial like adding additional columns is a solved problem.

Adding additional columns has always been trivial. What is not is the 98% of other things migrations do. Managing the version of the schema, applying ups in order, executing downs correctly, handling fk references. It’s not necessarily the fault of the migrations frameworks themselves, of which many exist in varying degrees of quality, but rather that the underlying problem of trying to morph a schema that is dependent on the underlying shape of the data is often a difficult problem armed with many footguns.

Re: YAGRI: You are gonna read it

#154

Post by WorldMaker »

rjbwork wrote: post #125
hypeatei wrote: post #120

Earlier quoted context omitted.

Temporal tables in SQL server fit this use-case[0], I think. 0: https://learn.microsoft.com/en-us/sql/relational-databases/t...

Available on postgres as an extension. It's a bit jank and doesn't have language integrated clauses like sql server.

Temporal Tables and their query language were formalized in ANSI SQL 2011 standard, so hopefully it is only a matter of time and resources to see it added into the postgres core directly.

Re: YAGRI: You are gonna read it

#155

Post by WorldMaker »

ajuc wrote: post #138
hypeatei wrote: post #120

Earlier quoted context omitted.

Temporal tables in SQL server fit this use-case[0], I think. 0: https://learn.microsoft.com/en-us/sql/relational-databases/t...

Oracle has flashback queries (SELECT ... AS OF timestamp). It's one of these things that are available but most people ignore it and implement it manually with created_at updated_at deleted_at columns etc. I think one reason for this is lack of support in ORMs and lack of standardization between RDBMSes.

Temporal Tables are ANSI Standard today (from SQL 2011). So far the closest to Standard is (surprisingly) Microsoft's SQL Server, but hopefully more of the DBs will pick up the standard sooner rather than later.

Re: YAGRI: You are gonna read it

#156

Post by crazygringo »

jeremyjh wrote: post #150
crazygringo wrote: post #139

Earlier quoted context omitted.

But what if it's not a technical decision? What if there are legal implications around data retention that it's not your job to be aware of? I've been parts of teams where features had to be totally thrown out and rebuilt because developers made big assumptions that turned out to be wrong, because they didn't think it was worth it to check with the product owner. Because they assumed it was only a "technical decision…

No. Just no. Put created_at, updated_at on every table. You are really grasping to find a problem with it, because there isn't one, and its been helpful in literally every job I've had for the last 28 years. Product owners don't do application support.

No, just no, back at you.

That's not all the article was suggesting. You're ignoring the other three fields the article says to "store on almost any table".

I'm not "grasping", I'm describing actual reality. While you're misrepresenting the article by cherry-picking the simplest fields, which aren't even always simple for the reasons I gave.

Re: YAGRI: You are gonna read it

#157

Post by rjbwork »

WorldMaker wrote: post #154
rjbwork wrote: post #125

Earlier quoted context omitted.

Available on postgres as an extension. It's a bit jank and doesn't have language integrated clauses like sql server.

Temporal Tables and their query language were formalized in ANSI SQL 2011 standard, so hopefully it is only a matter of time and resources to see it added into the postgres core directly.

Indeed...it's why I still pick MS SQL for certain projects.

Re: YAGRI: You are gonna read it

#158

Post by mike22 »

bayindirh wrote: post #133
dogleash wrote: post #132

Earlier quoted context omitted.

Oh, I thought you were disagreeing with hamandcheese's point that every little decision doesn't need to go through a product owner before anything happens.

No, not at all. by "the book", I meant regulations, not the management. :)

But it happens to be that product managers know (or at least about) and keep tabs on the relavent regulatory environment. I think it’s not scalable if every SWE in the team is going to legal to understand things. Like why we actully do need to hard delete data when customers click the Delete button.

Re: YAGRI: You are gonna read it

#159

Post by yellowapple »

Hasnep wrote: post #57
shaunofthedev wrote: post #26

The perils of UI design wagging the horse. I like the heuristics described here. However if these things aren't making it into a product spec where appropriate, then I smell some dysfunction that goes beyond what's being stored by default. Product need (expressed as spec, design, etc) should highlight the failure cases where we would expect fields like these to be surfaced. I'd hope that any given buisness shouldn't…

"Wagging the horse" is a great turn of phrase, better than "putting the cart before the dog."

Good ol' malaphors. I'm quite fond of "the cat's out of Pandora's box", "cross my I's and dot my T's", and "measure once cut twice".

Re: YAGRI: You are gonna read it

#160

Post by bayindirh »

mike22 wrote: post #158
bayindirh wrote: post #133

Earlier quoted context omitted.

No, not at all. by "the book", I meant regulations, not the management. :)

But it happens to be that product managers know (or at least about) and keep tabs on the relavent regulatory environment. I think it’s not scalable if every SWE in the team is going to legal to understand things. Like why we actully do need to hard delete data when customers click the Delete button.

If you force every SWE to go to legal for every technical decision, or ask for permission, it's not scalable, yes. On the other hand, if the team is in the long haul of developing this kind of regulated applications, the knowledge will get accumulated over time, and it'll trickle down from product managers to seniors to juniors.

This is the kind of tribal knowledge you want to spread among a development team, and if a collaborative document of "Why it's done this way" can be propped up with pointers to relevant sections of the regulation, it'd be a very good thing.

Not unlike NASA's global Lessons Learnt document.

Post reply on HN