41–50 of 58 posts
I wrote[0] about a method I've used with success in the past. Essentially you use the previous version's hash as the name for the next version. The benefit is that merging the work of multiple developers is easier. I ran the process manually when I did it and would love to hear if someone writes a script that makes the process easier. [0]: https://medium.com/@clord/for-migration-of-schemas-use-versi...
Earlier quoted context omitted.
> there is no concept of a "bug" in your data (so patches are meaningless). Is there not? Lets say that you're changing the data in your database/data structure from state X to state Y. This involves transforming the data in some tables/data structures from the old structure to the new. Lets say that you do this and it's all fine, the upgrade goes great. But then you discover there's a problem with the data upgrade.…
Any change you make to the schema will be a breaking change.
Earlier quoted context omitted.
> there is no concept of a "bug" in your data (so patches are meaningless). Is there not? Lets say that you're changing the data in your database/data structure from state X to state Y. This involves transforming the data in some tables/data structures from the old structure to the new. Lets say that you do this and it's all fine, the upgrade goes great. But then you discover there's a problem with the data upgrade.…
Any change you make to the schema will be a breaking change.
When transforming the schema, you frequently have associated changes that you apply to transform the data from one form (in the 'before' schema) to another (in the 'after' schema). These transformations are code that can have bugs like any other.
In cases like these you can have data that is in the right format, but isn't correct, and can need a second change (to the data only) to correct it.
Great question - we invented a system for this at Snowplow, called SchemaVer: http://snowplowanalytics.com/blog/2014/05/13/introducing-sch... SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless). We have hundreds of companies actively using SchemaVer via the Snowplow ( https://github.com/snowplow/snowplow/ ) and Iglu ( https://github.com/snowplow/iglu/…
I think you have a certain amount of fuzziness around the idea of an "interaction" with the data. It would probably help to think about compatibility and breaking changes in terms of reads vs. writes in order to get the determinism you're looking for and better alignment with SemVer.
That is, if a client using the previous schema can still do reads and writes without the data being invalid, you have forward compatibility, and this qualifies as an PATCH.
If a client using the previous schema can still do reads against the new schema without the data being invalid but not writes, that would qualify as a MINOR change.
(Aside: write-but-not-read compatible changes are possible, but are uncommon in practice)
A change that can prevent a client using the old schema from doing valid reads against the new schema (eg. a column is renamed or removed) would be a MAJOR change.
Thoughts?
Check out Pachyderm [0]. It supports distributed, version-controlled data storage. The API is very Git-like: you modify data by making commits. [0] https://github.com/pachyderm/pachyderm
CQRS and event sourcing is an approach where all data changes are versioned. You can re-create earlier states of the data by processing a portion of the event log.
I was thinking more along the lines of a versioned data "release".
Earlier quoted context omitted.
I am pretty familiar with several, such as Alembic, South, Django Migrations, etc. I'm not just interested in versioning schema changes, but data changes as well.
There are database migration tools like Liquibase/datical and, I believe, Flyway that let you specify migrations not just of schema changes (DDL) but also data changes (DML). I have used Liquibase in production for handling data changes for small-scale data (in megabytes) for dev/test/mock data and/or values in preconfigured application-controlling tables, but I wouldn't use it for gigabytes of transactional data.
In Data warehouse context, it is often managed by one of the Slowly Changing Dimension management techniques. https://en.m.wikipedia.org/wiki/Slowly_changing_dimension
In Data warehouse context, it is often managed by one of the Slowly Changing Dimension management techniques. https://en.m.wikipedia.org/wiki/Slowly_changing_dimension
Shameless plug: https://github.com/chop-dbhi/scds (stands for slowly changing dimensions store). A prototype, but versions individual records and tells you what changed if anything. Also there is preliminary support for JSON schema.