Live data from Hacker News

Versioning data in Postgres? Testing a Git like approach

specfy.io

11–20 of 92 posts

Re: Versioning data in Postgres? Testing a Git like approach

#11
Creates lots of complexity and Performance issues. I agree using PostgreSQL Extension for Temporal tables is a better idea.

I wish PostgreSQL would allow something as Oracle Flashback feature plus allowing to control how far history do you want to keep for specific tables, as typically only part of your data needs full auditing.

Re: Versioning data in Postgres? Testing a Git like approach

#14

Better idea: just install the Postgres extension that enables ISO SQL Temporal Tables, just like what MSSQL and MariaDB have had for 8 years now: https://pgxn.org/dist/temporal_tables/

author here: Yes I agree, and I mention it at the end. However it's not possible to install the extension everywhere, for example it's not available in GCP Cloud SQL unfortunately

Re: Versioning data in Postgres? Testing a Git like approach

#15

Better idea: just install the Postgres extension that enables ISO SQL Temporal Tables, just like what MSSQL and MariaDB have had for 8 years now: https://pgxn.org/dist/temporal_tables/

Does this robustly support altering tables? What happens if a column is deleted - does it delete historical data, or mark the history column as nullable? As the OP article notes/hints, there are all sorts of performance and regulatory considerations around this.

Separately though, it's great to see that there's still people trying to solve this problem at the extension level for Postgres. I was sad to discover that Postgres for a very long time had an official extension for time travel - https://www.postgresql.org/docs/11/contrib-spi.html#id-1.11.... - only to realize that it had been removed in https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... - with a note that "it's easy to implement this separately" but no real hints that anyone was taking up the mantle.

As a simpler alternative, if you're only modifying data via Django and can tolerate some unreliability, https://django-simple-history.readthedocs.io/en/latest/ can get you much of the way there without a need for extensions. We've built on top of it with some domain-specific modifications to visualize and filter diffs, giving us at least a best-efforts audit-esque log system.

Re: Versioning data in Postgres? Testing a Git like approach

#17

Better idea: just install the Postgres extension that enables ISO SQL Temporal Tables, just like what MSSQL and MariaDB have had for 8 years now: https://pgxn.org/dist/temporal_tables/

Does MySQL have temporal? I thought only MariaDB implemented this.

https://mariadb.com/kb/en/temporal-tables/

A quick Google turns up nothing for MySQL temporal.

Re: Versioning data in Postgres? Testing a Git like approach

#18
post #15

Better idea: just install the Postgres extension that enables ISO SQL Temporal Tables, just like what MSSQL and MariaDB have had for 8 years now: https://pgxn.org/dist/temporal_tables/

Does this robustly support altering tables? What happens if a column is deleted - does it delete historical data, or mark the history column as nullable? As the OP article notes/hints, there are all sorts of performance and regulatory considerations around this. Separately though, it's great to see that there's still people trying to solve this problem at the extension level for Postgres. I was sad to discover that P…

I wrote a guide and simple experimental setup for this extension: https://tembo.io/blog/table-version-history

Re: Versioning data in Postgres? Testing a Git like approach

#19
https://neon.tech/docs/introduction/branching is exactly what the author is looking for and is the best way to do highly efficient point-in-time versioning of Postgres data I have seen so far.

I deployed NeonDB in a large enterprise client in Q2 of this year and, yes, the initial data migration is the hard part. We added 2x100gbe network cards directly on the VMware hosts where the data was so the migration to the Kubernetes cluster running NeonDB took hours instead of weeks.

Re: Versioning data in Postgres? Testing a Git like approach

#20

Better idea: just install the Postgres extension that enables ISO SQL Temporal Tables, just like what MSSQL and MariaDB have had for 8 years now: https://pgxn.org/dist/temporal_tables/

Does MySQL have temporal? I thought only MariaDB implemented this. https://mariadb.com/kb/en/temporal-tables/ A quick Google turns up nothing for MySQL temporal.

My mistake, sorry; I meant MariaDB. I've edited my post.
Post reply on HN