1–10 of 58 posts
Is your data versioned? How?
I've seen three schemes used in production:
- single number versioning (V1->V2->V3->V4) - this is the scheme Android's sqlite helper insists on https://developer.android.com/reference/android/database/sql...
- full version numbers, but well defined upgrade tracks enforced in code.
- token based (e.g. a list of upgrades that have been applied)
What I have done in the past is to prefer lazy migrations (migrate once you access an old record), rather than migrating everything in batch.
Also a good idea to archive data that hasn't been used, moving it outside your primary database.
[0]: https://medium.com/@clord/for-migration-of-schemas-use-versi...
Do you mean schema, not data? I've seen three schemes used in production: - single number versioning (V1->V2->V3->V4) - this is the scheme Android's sqlite helper insists on https://developer.android.com/reference/android/database/sql... - full version numbers, but well defined upgrade tracks enforced in code. - token based (e.g. a list of upgrades that have been applied)
We do have a hash of all of the contents of the table, but the schema "version" is just the list of migrations that have been run. We run many different instances of our applications on several different "versions" and from time to time pull individual migrations back into earlier releases for hotfixes etc. The overall version is the instance, environment, and this hash.
Practically speaking, when we're handling the files ourselves though, it's instance, environment and backup date.
There are edge cases and it's not perfect (around ordering mostly), but it's incredibly rare for us to run into issues with this schema / data migration approach. We've found that even with a bunch of environments and a couple thousand migrations, data versioning isn't that serious a problem for us.
Do you mean schema, not data? I've seen three schemes used in production: - single number versioning (V1->V2->V3->V4) - this is the scheme Android's sqlite helper insists on https://developer.android.com/reference/android/database/sql... - full version numbers, but well defined upgrade tracks enforced in code. - token based (e.g. a list of upgrades that have been applied)
That is, imagine that after a version of the data is released it increases 10X, but the schema doesn't change. That might still be worth making a release for, right?
Typically what I see in the wild are data dumps on some sort of semi-regular schedule such as weekly, monthly, or annually, "versioned" with the timestamp or some derivative thereof (eg. "August 2016").
This is much better than nothing, but just as a source dump for each release of an application or other software in a tarball is not as useful as being able to check out a tagged release from a repo, so too versioned data releases that can be checked out with a tag from a repo would be more helpful that a data tarball.
I sometimes do it within the same table. It's a bit of a hack. What you want is a column called revision_id (something to point back to the master record). You'd want a state column to track if it's a draft, revision, active, etc. You might also want a sequence column too, so that when you apply drafts, you can compare if the draft can modify the master record (if sequence column doesn't match, you prevent the draft…
I wrote[0] about a method I've used with success in the past. Essentially you use the previous version's hash as the name for the next version. The benefit is that merging the work of multiple developers is easier. I ran the process manually when I did it and would love to hear if someone writes a script that makes the process easier. [0]: https://medium.com/@clord/for-migration-of-schemas-use-versi...