Ask HN: How do you version your data? • Hacker News Board

Re: Ask HN: How do you version your data?

#11

Post by webmaven » Mon, Feb 20, 2017, 3:49 AM UTC

px1999 wrote: post #6

Our migrations have unique datetime stamps. Stamps and migration names go into a table. We do have a hash of all of the contents of the table, but the schema "version" is just the list of migrations that have been run. We run many different instances of our applications on several different "versions" and from time to time pull individual migrations back into earlier releases for hotfixes etc. The overall version is…

So, the versioning of the data is basically by the backup date (plus pointers to the relevant schema, etc.), correct?

Re: Ask HN: How do you version your data?

#12

Post by kchoudhu » Mon, Feb 20, 2017, 3:55 AM UTC

ALL migrations insert major.minor.patch to the database. Migrations are committed to the database, and can be run anytime; they each have check code at the beginning to compare against what's in the database before executing the payload. Migrations won't execute if the version is semantically less or too far ahead from what's in the database.

It's worked reasonably well so far.

Re: Ask HN: How do you version your data?

#13

Post by derekchiang » Mon, Feb 20, 2017, 3:58 AM UTC

Check out Pachyderm [0]. It supports distributed, version-controlled data storage. The API is very Git-like: you modify data by making commits.

[0] https://github.com/pachyderm/pachyderm

Re: Ask HN: How do you version your data?

#14

Post by webmaven » Mon, Feb 20, 2017, 3:58 AM UTC

badminton1 wrote: post #3

I suggest you look into schema migration tools. What I have done in the past is to prefer lazy migrations (migrate once you access an old record), rather than migrating everything in batch. Also a good idea to archive data that hasn't been used, moving it outside your primary database.

I am pretty familiar with several, such as Alembic, South, Django Migrations, etc.

I'm not just interested in versioning schema changes, but data changes as well.

Re: Ask HN: How do you version your data?

#15

Post by [deleted] » Mon, Feb 20, 2017, 4:09 AM UTC

[deleted]

Re: Ask HN: How do you version your data?

#16

Post by rishabhsagar » Mon, Feb 20, 2017, 4:17 AM UTC

In Data warehouse context, it is often managed by one of the Slowly Changing Dimension management techniques.

https://en.m.wikipedia.org/wiki/Slowly_changing_dimension

Re: Ask HN: How do you version your data?

#17

Post by jsudhams » Mon, Feb 20, 2017, 4:37 AM UTC

Have seen two ways of doing in large organizations 1. Use tools like Optim (i think ibm) 2. Export CSV data, and export schema and drop it to Hadoop data store

Option 2 is preferred for very long time restore say after 15 years as there may not be a way to restore to Oracle/MS Products which may not have run time ecosystem after 15 years.

Re: Ask HN: How do you version your data?

#18

Post by j_s » Mon, Feb 20, 2017, 5:38 AM UTC

This project seems to be making progress for WordPress: https://versionpress.net/

Re: Ask HN: How do you version your data?

#19

Post by [deleted] » Mon, Feb 20, 2017, 7:02 AM UTC

[deleted]

Re: Ask HN: How do you version your data?

#20

Post by hbcondo714 » Mon, Feb 20, 2017, 7:18 AM UTC

On a somewhat related note, I have an API that returns time series data. The API itself is versioned but not the data when it is added to since the endpoint is the same. How would I go about notifying consumers of the API that the data has been updated and they should call the API to get the latest data? Would versioning the data help with this?