Migrations and Future Proofing
github.com
Migrations and Future Proofing
1–10 of 13 posts
Re: Migrations and Future Proofing
#2This is a really good write-up.
In consulting and mentoring on this topic, I've found a lot of engineers push back against how "dirty" it is to have multiple copies of the data around in different formats. It feels wrong to not have a single, authoritative data format at any given instant. If the idea is to change the column type, why not just `ALTER TABLE ... ALTER COLUMN` instead of `ALTER TABLE ... ADD`?
But if you think about it, excepting trivial cases, once you're migrating data, there are parallel realities at least for the duration of the migration and deployment. It's not a question of whether you create divergence by versioning/staging (in some fashion) your data. It's a question of whether you manage the divergence and convergence of the parallel realities that already exist as part of a migration. If you don't, you either incur downtime or risk data corruption.
One big win here is that, by being disciplined about your code and data changes, you can cleanly separate deployment from release. You can deploy a feature but have it disabled or only enabled for a subset of users. Releasing a feature means enabling its feature flag, not orchestrating a set of migrations, replications, and deployments.
Re: Migrations and Future Proofing
#3 When you introduce a new API endpoint or format
for data at rest, think hard
If you're designing an experimental server-side
feature, see if you can store the data off to the
side (e.g., in a different location, rather than
together with currently critical data) so you can
just delete it if the experiment fails rather than
being saddled with this data forever without a
huge migration project.
Re: Migrations and Future Proofing
#4When you introduce a new API endpoint or format for data at rest, think hard Yup. I've added columns where I've used a datetime where a date would have sufficed and then regretted it later once tons of data was already in the table. Or added a varchar(255) and only later realized that that wasn't big enough. Sometimes the wrongness of a type only becomes clear down the road. If you're designing an experimental server…
Why is it hard to drop an extra column from a database, or to change its type? ALTER TABLE...
Re: Migrations and Future Proofing
#5How does Erlang deal with these problems? It often touts minimal downtime and the ability to run updates to your code while it's running.
I think that means you can have Process V1 and Process V2 running on the same server simultaneously. If they read from the same database, won't you run into issues?
Re: Migrations and Future Proofing
#6How does Erlang deal with these problems? It often touts minimal downtime and the ability to run updates to your code while it's running. I think that means you can have Process V1 and Process V2 running on the same server simultaneously. If they read from the same database, won't you run into issues?
If I understand correctly, this is the point of not modifying the existing data. Current processes will continue to work, as you aren't changing them. New processes will rely on new data (from possibly new stores) such that they shouldn't change what is current.
I could be misreading, of course.
Re: Migrations and Future Proofing
#7When you introduce a new API endpoint or format for data at rest, think hard Yup. I've added columns where I've used a datetime where a date would have sufficed and then regretted it later once tons of data was already in the table. Or added a varchar(255) and only later realized that that wasn't big enough. Sometimes the wrongness of a type only becomes clear down the road. If you're designing an experimental server…
Why is it hard to drop an extra column from a database, or to change its type? ALTER TABLE...
It a) takes time, and b) means current code can not possibly work. Worse, it c) means canceling a rollout is now a complicated process, due to b.
Re: Migrations and Future Proofing
#8Author here, would love feedback on this, and also happy to answer any questions.
Re: Migrations and Future Proofing
#9When I was at Google this was the single worst problem we had in engineering, at least in terms of engineer-hours consumed. We came up with a bunch of solutions, a few of which (like protobufs) are open-sourced and many of which are just in the heads of the engineers who did them, but there's unfortunately no general solution to the problem. Sometimes I dream about a programming language that has thought through all these issues and includes "evolvability" as a first-class design constraint, but oftentimes these problems show up in multi-process situations where you may be using multiple programming languages.
Re: Migrations and Future Proofing
#10At least for the server size, entity framework partially solves this with code first migrations. Using the code first model you can define your data types in code then have entity framework generate the appropriate sql code. If you change your data structure down the road you can autogenerate a migration that changes the database from one version to another. If you deploy a version that is a few migrations ahead then it will execute the proper migrations one after the other.
For the client side it's usually a good idea to specify a versioning relationship between server and client. AWS, for example, you request the API version you want to use: http://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGu...