Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

YAGRI: You are gonna read it

scottantipa.com

Post reply on HN
Print view View original

141–150 of 161 posts

Re: YAGRI: You are gonna read it

#141

Post by PaulHoule »

Funny I've been developing an adaption layer that implements the functionality that I use in

https://docs.python-arango.com/en/main/

over tables in Postgres that has a PRIMARY _key and a JSONB document field. The issue is that I have a number of prototypes I've developed with arangodb but the license is awful and I don't feel like I can either open source or commercialize any of them until I'm running on an open source database.

It's a fun project because I don't need to support everything in python-arango, in fact I don't even need to support 100% of the features I use because I am free to change my applications. Also it's a chance to make the library that I really want to use so already it has real integer and UUID primary keys.

I just added a feature to have the library manage _created and _updated fields not just because I thought it was good in general but it was a feature I needed for a particular application, a crawler that fetches headlines from the HN API. I want to fetch headlines right away so I can avoid submitting duplicates but I also want accurate counts of how many votes and comments articles got and that involves recrawling again in two weeks. Of course _created and _updated are helpful for that.

Re: YAGRI: You are gonna read it

#142

Post by koakuma-chan »

code_biologist wrote: post #22
mkhalil wrote: post #7

Event-sourcing solves this. And with how cheap storage is, it should be more prevalent in the industry. IMO the biggest thing holding it back is that there isn't a framework that's plug-and-play (say like Next.js is to React) that provides people with that ability. I've been working on one in Typescript (with eventual re-writes in other langs. like Rust and Go), but it's difficult even coming up with conventions.

Event sourcing is an expensive solution and I don't mean from a storage perspective — it burns engineering cognitive horsepower quickly on things that don't matter. Do it if you're in finance or whatever. Having been burned by my own "let's event source" impulse on data change tracking systems, I now prefer less sophisticated solutions. Figuring out how to deal with slow projections, watching a projection rebuild go…

Why not keep the projection always updated?

Re: YAGRI: You are gonna read it

#143

Post by r7n »

We did exactly this when designing StatleyDB. We realized there are a set of metadata fields[0] that almost everyone needs when modeling data. Our approach takes this a step further in that we always track these fields transparently so if you forget to add them to your schema initially for any reason, that's okay, you can always add them later!

[0] https://docs.stately.cloud/schema/fields/#metadata-fields

Re: YAGRI: You are gonna read it

#144

Post by escot »

crazygringo wrote: post #60

These are not decisions that should be taken solely by whoever is programming the backend. They need to be surfaced to the product owner to decide. There may very well be reasons pieces of data should not be stored. And all of this adds complexity, more things to go wrong. If the product owner wants to start tracking every change and by who, that can completely change your database requirements. So have that conversa…

In a good work environment you’ll have some telepathy with your product people and you wont have to communicate every minor change, which wastes time. Its similar to implementing a UI design — you and the designer will develop a relationship where you dont need to communicate every pixel, you fill in the blanks and show results, repeat.

Re: YAGRI: You are gonna read it

#145

Post by ajuc »

kamatour wrote: post #131

Why can't databases just remember stuff we delete, like a trash can?

They do if you tell them. Both Oracle, mssql and postgresql have this implemented. I only used it in Oracle - it's called flashback queries - you do SELECT ... AS OF timestamp and you can see state of the table from the past.

It needs to be enabled tho, and it has some problems (for example you need to remove data for legal reasons sometimes).

Reading this in thread is very frustrating with people reinventing the wheel and asking why it wasn't invented already:)

Re: YAGRI: You are gonna read it

#146

Post by mastax »

hypeatei wrote: post #120
KronisLV wrote: post #109

I don't get why all of the big RDBMSes (PostgreSQL, MariaDB/MySQL, SQL Server, Oracle, ...) don't seem to have built in support for soft deletes up front and center? CREATE TABLE ... WITH SOFT DELETES Where the regular DELETE wouldn't get rid of the data for real but rather you could query the deleted records as well, probably have timestamps for everything as a built in low level feature, vs having to handle this wi…

Temporal tables in SQL server fit this use-case[0], I think. 0: https://learn.microsoft.com/en-us/sql/relational-databases/t...

Also System-Versioned Tables in MariaDB: https://mariadb.com/kb/en/system-versioned-tables/

Re: YAGRI: You are gonna read it

#147

Post by perrygeo »

KronisLV wrote: post #109

I don't get why all of the big RDBMSes (PostgreSQL, MariaDB/MySQL, SQL Server, Oracle, ...) don't seem to have built in support for soft deletes up front and center? CREATE TABLE ... WITH SOFT DELETES Where the regular DELETE wouldn't get rid of the data for real but rather you could query the deleted records as well, probably have timestamps for everything as a built in low level feature, vs having to handle this wi…

The whole model of RDBMS is based on mutable tuples; soft deletes don't make much sense as an intrinsic part of that model. If you want soft deletes, you create an application layer or use a different data model.

Most of the time if you want "soft deletes", you really want an immutable log so that you time travel to any point in the history. XTDB and Datomic are worth looking at if you want to solve the problem at the data model level.

Re: YAGRI: You are gonna read it

#149

Post by mrkeen »

dml2135 wrote: post #24
zie wrote: post #18

Earlier quoted context omitted.

If you have a good audit log, it really doesn't matter. You can always restore it if need be. If you have no audit log(or a bad one), like lots of apps, then you have to care a lot. Personally, I just implement a good audit log and then I just delete with impunity. Worst case scenario, someone(maybe even me) made a mistake and I have to run undo_log_audit() with the id of the audit log entry I want to put back. Nearl…

Can you share more about what makes a good audit log? My company doesn’t currently have one and I’m a little lost on where to start. Should this be at the application code level, or the ORM, or the database itself?

I once worked in a small VB6-based team - you can probably guess the attitudes and surrounding tech were just as out of date.

I tried to push for using svn, rather than just making copies of our source code folders and adding dates to them.

My manager allowed me to use svn, but to make sure I also did things the proper way by making copies of the source code folders.

That's the current level of discourse around audit logs. Write down what happened using your data tables ... but write down what really happened in the audit logs.

At some point you should just lean into putting audit logs first (just like developers reach for the git first).

Re: YAGRI: You are gonna read it

#150

Post by jeremyjh »

crazygringo wrote: post #139
Extasia785 wrote: post #112

Earlier quoted context omitted.

This entirely depends on the company culture. I worked in teams where every small decision is in the hand of the PO and I've worked in teams where a software engineer is a respected professional enabled to make their own technical decisions. I found the second option to create higher quality software faster. Also not sure what you mean by additional effort? Created_at, updated_at or soft-deletes are part of most prop…

But what if it's not a technical decision? What if there are legal implications around data retention that it's not your job to be aware of? I've been parts of teams where features had to be totally thrown out and rebuilt because developers made big assumptions that turned out to be wrong, because they didn't think it was worth it to check with the product owner. Because they assumed it was only a "technical decision…

No. Just no. Put created_at, updated_at on every table. You are really grasping to find a problem with it, because there isn't one, and its been helpful in literally every job I've had for the last 28 years. Product owners don't do application support.
Post reply on HN