You might want to check Datomic out. They have a solution for your problem: http://www.datomic.com
Ask HN: How do you version your data?
31–40 of 58 posts
Re: Ask HN: How do you version your data?
#32In Data warehouse context, it is often managed by one of the Slowly Changing Dimension management techniques. https://en.m.wikipedia.org/wiki/Slowly_changing_dimension
Shameless plug: https://github.com/chop-dbhi/scds (stands for slowly changing dimensions store). A prototype, but versions individual records and tells you what changed if anything. Also there is preliminary support for JSON schema.
Re: Ask HN: How do you version your data?
#33Depends on what it is, either semver for things where backward compatibility is necessary or just "version 1, version 2 etc." where a new version is a big enough change you should consult the documentation to know what is going on.
Re: Ask HN: How do you version your data?
#34I use protobuf schema evolution. It's the same approach that Google use (naturally), and Facebook (with Thrift), among others.
Re: Ask HN: How do you version your data?
#35Great question - we invented a system for this at Snowplow, called SchemaVer:
http://snowplowanalytics.com/blog/2014/05/13/introducing-sch...
SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless).
We have hundreds of companies actively using SchemaVer via the Snowplow (https://github.com/snowplow/snowplow/) and Iglu (https://github.com/snowplow/iglu/) projects.
Re: Ask HN: How do you version your data?
#36Great question - we invented a system for this at Snowplow, called SchemaVer: http://snowplowanalytics.com/blog/2014/05/13/introducing-sch... SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless). We have hundreds of companies actively using SchemaVer via the Snowplow ( https://github.com/snowplow/snowplow/ ) and Iglu ( https://github.com/snowplow/iglu/…
+1 for Snowplow's approach. If you have an app out there in the wild (with autoupgrades off for at least 10% of devices), there isn't such a thing as "we'll do a hard cutoff for our tracking data". Their way, the data is versioned in self-describing contexts so they are separated from JavaScript tracker to database (two different tables), and you can write an ongoing migration between the tables. Very helpful.
Re: Ask HN: How do you version your data?
#37Great question - we invented a system for this at Snowplow, called SchemaVer: http://snowplowanalytics.com/blog/2014/05/13/introducing-sch... SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless). We have hundreds of companies actively using SchemaVer via the Snowplow ( https://github.com/snowplow/snowplow/ ) and Iglu ( https://github.com/snowplow/iglu/…
> there is no concept of a "bug" in your data (so patches are meaningless).
Is there not? Lets say that you're changing the data in your database/data structure from state X to state Y. This involves transforming the data in some tables/data structures from the old structure to the new.
Lets say that you do this and it's all fine, the upgrade goes great. But then you discover there's a problem with the data upgrade.
While you have transformed the data into the new format, it's not been done right. So you actually need a second data change to ensure that your data upgrade is semantically equivalent to the data that went before it, even though the data conforms to your new schema.
Would that not count as a bug in your data?
Re: Ask HN: How do you version your data?
#38I suggest you look into schema migration tools. What I have done in the past is to prefer lazy migrations (migrate once you access an old record), rather than migrating everything in batch. Also a good idea to archive data that hasn't been used, moving it outside your primary database.
I am pretty familiar with several, such as Alembic, South, Django Migrations, etc. I'm not just interested in versioning schema changes, but data changes as well.
There are database migration tools like Liquibase/datical and, I believe, Flyway that let you specify migrations not just of schema changes (DDL) but also data changes (DML).
I have used Liquibase in production for handling data changes for small-scale data (in megabytes) for dev/test/mock data and/or values in preconfigured application-controlling tables, but I wouldn't use it for gigabytes of transactional data.
Re: Ask HN: How do you version your data?
#39There are several tools out there for doing this, if you're storing your data in databases.
SQL Server has Master Data Services. There's also Talend. There are other master data management tools available.
Depending on how you use your data, there are tools for improving data quality that tie in with those as well.
Re: Ask HN: How do you version your data?
#40Great question - we invented a system for this at Snowplow, called SchemaVer: http://snowplowanalytics.com/blog/2014/05/13/introducing-sch... SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless). We have hundreds of companies actively using SchemaVer via the Snowplow ( https://github.com/snowplow/snowplow/ ) and Iglu ( https://github.com/snowplow/iglu/…
> there is no concept of a "bug" in your data (so patches are meaningless). Is there not? Lets say that you're changing the data in your database/data structure from state X to state Y. This involves transforming the data in some tables/data structures from the old structure to the new. Lets say that you do this and it's all fine, the upgrade goes great. But then you discover there's a problem with the data upgrade.…
Any change you make to the schema will be a breaking change.