Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Versioning data in Postgres? Testing a Git like approach

specfy.io

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Versioning data in Postgres? Testing a Git like approach

#81

Post by _a_a_a_ »

GordonS wrote: post #65
_a_a_a_ wrote: post #28

Earlier quoted context omitted.

Using hints is typically a sign of failure. The DB optimiser should typically not need them .

In a perfect world, I wouldn't disagree; but we don't live in a perfect world where everything always goes your way. Sometimes the planner needs a little help, and that's OK.

I did say 'typically'! Yes, agreed, but it should rarely need help. Also hints are oh so often jammed in by newbies. I strongly get the impression use of hints is inversely correlated with experience/knowledge.

Re: Versioning data in Postgres? Testing a Git like approach

#82

Post by GordonS »

_a_a_a_ wrote: post #81
GordonS wrote: post #65

Earlier quoted context omitted.

In a perfect world, I wouldn't disagree; but we don't live in a perfect world where everything always goes your way. Sometimes the planner needs a little help, and that's OK.

I did say 'typically'! Yes, agreed, but it should rarely need help. Also hints are oh so often jammed in by newbies. I strongly get the impression use of hints is inversely correlated with experience/knowledge.

I'm a greybeard you cheeky scamp ;)

I very rarely used hints in my more database'y days - but every so often, they were needed, and often to tell the database something that seemed screamingly obvious to me.

I'm not sure where you got the notion that hints were something noobs would be working with? At least, that certainly wasn't my experience.

Re: Versioning data in Postgres? Testing a Git like approach

#83

Post by amarshall »

> If you have rewrote a file 999 times, there is 999 times copy of this file in the object storage. That's why it can be slow to clone a git repository.

Sort of. While Git’s canonical storage is indeed not diffs, it uses delta compression to, well, store diffs as a storage and transit optimization https://git-scm.com/book/en/v2/Git-Internals-Packfiles

Re: Versioning data in Postgres? Testing a Git like approach

#84

Post by _a_a_a_ »

GordonS wrote: post #82
_a_a_a_ wrote: post #81

Earlier quoted context omitted.

I did say 'typically'! Yes, agreed, but it should rarely need help. Also hints are oh so often jammed in by newbies. I strongly get the impression use of hints is inversely correlated with experience/knowledge.

I'm a greybeard you cheeky scamp ;) I very rarely used hints in my more database'y days - but every so often, they were needed, and often to tell the database something that seemed screamingly obvious to me. I'm not sure where you got the notion that hints were something noobs would be working with? At least, that certainly wasn't my experience.

We are in strong agreement, and the phrasing and measure of your post made it pretty obvious you are knowledgeable (another inverse correlation: absolute opinions launched with violence also seemed to be the realm of newbies; and yours was eminently not).

> I very rarely used hints in my more database'y days

evidence thereof!

> I'm not sure where you got the notion that hints were something noobs would be working with? At least, that certainly wasn't my experience.

It's been very, very much my experience. I worked in one place where nolock hints were applied everywhere, without them realising that READ UNCOMMITTED trans iso even existed, so had to show them that, and show them it could be specified at the application layer (not in the SQL). In another recent case I came across their use and I'm damn sure the person who used it didn't actually know what it did. Yeah, I've seen far too much hinting in my career.

Re: Versioning data in Postgres? Testing a Git like approach

#85

Post by zerbinxx »

nikita wrote: post #33
devaiops9001 wrote: post #19

https://neon.tech/docs/introduction/branching is exactly what the author is looking for and is the best way to do highly efficient point-in-time versioning of Postgres data I have seen so far. I deployed NeonDB in a large enterprise client in Q2 of this year and, yes, the initial data migration is the hard part. We added 2x100gbe network cards directly on the VMware hosts where the data was so the migration to the Ku…

Neon CEO here. Did you deploy Neon on prem?

Hi Neon CEO. I’m curious if users have deployed Neon as a (selective) audit-logging tool for configuration which might need to be edited or rolled back from time to time. Think configuration-driven/event-driven systems where an update to a single database-stored config can cause other downstream applications to start behaving mysteriously. Going to scour the website some more!

Re: Versioning data in Postgres? Testing a Git like approach

#86

Post by nikita »

zerbinxx wrote: post #85
nikita wrote: post #33

Earlier quoted context omitted.

Neon CEO here. Did you deploy Neon on prem?

Hi Neon CEO. I’m curious if users have deployed Neon as a (selective) audit-logging tool for configuration which might need to be edited or rolled back from time to time. Think configuration-driven/event-driven systems where an update to a single database-stored config can cause other downstream applications to start behaving mysteriously. Going to scour the website some more!

We have customers using Neon as a time machine b/c you can create a branch at a point in time. Is this what you need?

Re: Versioning data in Postgres? Testing a Git like approach

#89

Post by zerbinxx »

nikita wrote: post #86
zerbinxx wrote: post #85

Earlier quoted context omitted.

Hi Neon CEO. I’m curious if users have deployed Neon as a (selective) audit-logging tool for configuration which might need to be edited or rolled back from time to time. Think configuration-driven/event-driven systems where an update to a single database-stored config can cause other downstream applications to start behaving mysteriously. Going to scour the website some more!

We have customers using Neon as a time machine b/c you can create a branch at a point in time. Is this what you need?

Essentially, yes. Between a few times per day and a few times per month I would like to create branches to store the “backup” copies of table-stored confs to compare against logs/events and see if configuration changes caused certain conditions to arise. In the case of a configuration error, rolling back to a prior state would be awesome.

Re: Versioning data in Postgres? Testing a Git like approach

#90

Post by naasking »

Rather than generic blobs, you can easily integrate versioning directly in your entity tables with only two extra fields:

https://higherlogics.blogspot.com/2015/10/versioning-domain-...

With this schema, you simply add an extra clause to each of your queries and they can simultaneously return the latest version or any past version.

Post reply on HN