Live data from Hacker News

Ask HN: How do you organize and manage database migrations?

news.ycombinator.com

31–40 of 61 posts

Re: Ask HN: How do you organize and manage database migrations?

#31
> but both are not completely free and seem proprietary

I don't mean to be "that guy", but unless your whole business's "thing" is no-proprietary-software, please try to pay for commercial licenses of good software. That money pays for bug fixes, security fixes, general maintenance, feature development, testing, support, and a host of other crap you probably don't want to have to do by yourself. If they provide a free tier, you're also supporting the free users.

Flyway is kind of the gold standard, I recommend it.

Re: Ask HN: How do you organize and manage database migrations?

#33
I've previously used some ORM-based migration tools, like Django ORM migrations (and when it was a separate tool, South), looked at others like Alembic. What I found is that they automate simplest cases, in turn making more complex cases even more complex.

So when I need to do some complex migration on a big database, I usually open postgres shell, open a transaction and I develop a migration like a code, in a REPL. And in case of these automatic tools, after I did that I have to go read their documentation, and port SQL to their syntax.

If I have junior members on the team who don't know SQL yet on a good level, easier tasks are automated by the tool, and for more complex task they completely lack skills developing a migration and have a much steeper wall to climb.

Another thing is that many tools have downgrade migrations. Downgrade migrations are a lie! How could I revert a migration that drops a NOT NULL column with some data? In case I really need to revert a migration I will write another forward migration. Which I did exactly zero times in more than ten years. So, writing downgrade migrations is a waste of time.

Another minor point is that sometimes migrations are running for a loooooong time. Not a second or two - it can be many hours. So I don't want this migration to start automatically during my deploy process. But I don't want to have it as a separate script, so on local dev installations these migrations just take the same `make migrate` route. It's much more convenient to take a part of a commented SQL and run it separately, than to take a part of a migration script written in Python and run it separately.

I'm using a Nomad tool https://pypi.org/project/nomad/ which is written in Python. I think any tool that supports plain SQL migrations, has dependencies between migrations and doesn't require to write downgrades would be acceptable.

Re: Ask HN: How do you organize and manage database migrations?

#36

Honestly? I don't know of any tool that handles database migrations as well or as easily as Ruby on Rails and ActiveRecord. I have literally spun up a barebones Rails app just to manage my schema and migrations for a completely separate Python/PostgreSQL project before. https://guides.rubyonrails.org/v5.2/active_record_migrations...

We use this for migrations on the daily..

https://pypi.org/project/yoyo-migrations/

Re: Ask HN: How do you organize and manage database migrations?

#37
Flyway is free open source software (licensed under the Apache License 2.0).

There are pro and enterprise licenses in case you have additional requirements. These licenses as I understand it are proprietary and hence non-free but the the community edition absolutely is free software.

I’ve been using the community edition in multiple projects for several years now and it works great, particularly in the context of Spring Boot applications.

It might not be just as polished and well-integrated as ActiveRecord is for Ruby on Rails but keep in mind Flyway is supposed to be framework-independent so it can’t be too tightly integrated.

That said, its integration with Spring Boot is really good. Just add it as a dependency and configuration and migrations during application startup will be taken care of automatically.

Re: Ask HN: How do you organize and manage database migrations?

#38
I wrote a tool for that in our company (in Java, but could be written in any language) like 10 years ago or so, which is still in heavy use as of today in multiple projects, migrating tens of thousands of (SQLServer and PostgreSQL) databases each year. Its key feature is probably the usage of a schema hash, which seems to be pretty unique among DB migration tools (at least the common open source equivalents to what I wrote don't seem to do this). It works like this:

- The tool can hash the current schema found in a live database that it's connected to

- It also knows hashes for all schemas ever created for the application (these are versioned and usually originate largely from an ORM generator, but can also mix generated parts with manually-added ones)

- In addition to knowing these "full schemas" with their respective versions, it knows forward (and optionally backward) migrations between the versions (these are entirely created by developers). Migrations usually perform DDL stuff, but may optionally also convert data either with pure SQL or procedurally (as migrations are effectively stored as Java code, they can do practically anything)

- When run against a database, it can either initialize it to any version of the software using one of the full schemas, or migrate it from any known schema version to any other version to which it can find a migration path (pathfinding is a cheap Dijkstra implementation on the migration graph). It automatically detects at which version a schema is by hashing it and comparing the hash with all known schema versions and their hashes (with the ability to fall back to metadata of past migrations also stored in the database in case two versions result in a hash collision). Before and after every migration step, the hash is built from the current state of the database and checked against the stored values for before and after the migration step in question, to ensure the step finds everything as expected and resulted in the correct final state.

- If a backward migration path can be built, migrating "down" is also possible (but I've never seen this actually used in practice, hence most teams today just skip writing backwards migration steps).

- The hashes are also used to prevent accidents that might happen if someone runs a migration against a database on which some person manually modified anything schema-relevant. In such case, the hash won't match any version, and the migration will not even start. Whatever manual modification has been done must be undone first in order to guarantee that the migration steps can find the exact state that they were written for and thus yield the expected results (this actually saved our asses multiple times).

- Finally, a variant of the tool was integrated in the CI process, which means that all migration steps currently checked in are tested against an actual database with each build and can fail the build if not working. Since the current schema can be generated from an object model, the tests also generate this automatically and check its hash against whatever the currently highest versioned migration would produce, failing the test if these are not equal. Effectively this results in developers being immediately notified if someone did change the database schema (usually indirectly via an object model change) but did not check in the necessary migrations to get old versions migrated upwards.

Re: Ask HN: How do you organize and manage database migrations?

#39
post #5

I've been using Dbmate for a few years now across our entire stack. It does the job perfectly. https://github.com/amacneil/dbmate

I can second Dbmate.

The process of a sql migration management is pretty simple and like other comments it's not hard to write something yourself which works well I just prefer to use something a little more battle tested in the deployment process.

Also it's a go app so you don't need to pull down a stupid amount of dependencies for it to work just pull from it's github release and mark it as executable.

Re: Ask HN: How do you organize and manage database migrations?

#40

Honestly? I don't know of any tool that handles database migrations as well or as easily as Ruby on Rails and ActiveRecord. I have literally spun up a barebones Rails app just to manage my schema and migrations for a completely separate Python/PostgreSQL project before. https://guides.rubyonrails.org/v5.2/active_record_migrations...

Ever tried EntityFramework in .Net ? It's amazingly easy and there are 2 ways:

- Database-First

- Code-First

Post reply on HN