Live data from Hacker News

Better Database Migrations in Postgres

craigkerstiens.com

51–60 of 89 posts

Re: Better Database Migrations in Postgres

#51

strong_migrations looks like a really useful tool. Are there similar ones for languages/frameworks besides Ruby/Rails?

Alembic is a standalone tool written in Python. Having spent years on rails migrations and then using Alembic, I can say that alembic is really brilliant. It behaves like git - it has a branching and merge model for Migrations (in case multiple people work on it simultaneously).

but it seems like the USP of strong_migrations is to recognise 'dangerous' operations in a migration and warn you about them so that you can rewrite it as a safer migration

Does alembic have anything similar? this would be useful for Python projects

Re: Better Database Migrations in Postgres

#52

OT: Postgres is an incredible piece of open source software, but the official admin UI pgAdmin is in a state of complete chaos. Any suggestions for an OSS replacement?

Postage – A fast replacement for pgAdmin | https://news.ycombinator.com/item?id=14884183 (Jul 2017)

> skrause: BigSQL maintains an LTS release that stays compatible with the most recent Postgres versions: https://www.openscg.com/bigsql/pgadmin3/

The discussion mentions a number of additional alternatives. Also: Postage itself is looking for a new maintainer.

Re: Better Database Migrations in Postgres

#53
post #35

Earlier quoted context omitted.

Sql Server with Management Studio can do something like it with the dacpak format. It calculates differences and what needs to be done.

We have a project at work using this dacpak stuff and tbh it's a nightmare to work with since the dacpaks themselves have to manually generated and distributed. It also appears to make the projects (in visual studio) unbelievably slow.

I'm with you that it makes the solution slower, but you can just selectively not load it, or use the new light loading feature.

You most certainly do not have to make them manually, that can be delegated to a build server. You can also use the dacpac to automatically run integration test and it can run some rudimentary upgrade tests vs previous versions of the dacpac.

Re: Better Database Migrations in Postgres

#54
> Gradually backfill the default value

Might I add a little warning from experience.

The "gradually" part is important to get right, since in postgres updates actually create new tuples and leave dead tuples behind, if you update too much at once two things might happen: you increase disk usage fast (worth considering how much room you have to work and adjust before if needed) and increase the amount of dead tuples fast.

I have experienced that having too much dead tuples on a table might impact the query planner and it may stop using some index completely before autovacuum does its job. Depending on the circumstance this may severely impact performance if an important query suddenly starts Seq Scanning a huge table.

This blog post has great practical details to consider before running such updates: https://www.citusdata.com/blog/2016/11/04/autovacuum-not-the... In particular at the time we had to adjust the "autovacuum_vacuum_scale_factor" for the table in order for the autovacuum process to kick in much sooner. Also there is a valuable query there to inspect how much dead tuples you have.

Re: Better Database Migrations in Postgres

#55
post #25

I really just want a deterministic (or sync-based) migration tool. The only two I'm aware of are Innovartis DBGhost and RedGate SQL Compare (though RedGate requires a license everywhere it runs, whereas DBGhost only requires a license to compile the package). This stems from my work years ago with databases in on-premise products. Customers would modify the database schema, causing migration "up" scripts to fail, and…

At Facebook we discovered this is especially important for large sharded environments. As you add more shards, the chances that a migration will fail on at least one of them (simply due to hardware failure) increases. A declarative approach, where you have a repo of CREATE statements that your tools can diff/sync automatically, is much easier to manage. You can have automation just retry in a loop until the shard converges on the desired final state.

> I've been looking for something similar for Postgres and MySQL/MariaDB without any luck

For MySQL, I wrote an open source tool to do this: http://github.com/skeema/skeema -- it gives you a repo-of-CREATE-statements approach to schema management, like you describe.

Skeema's CLI is inspired by Git (in terms of paradigm for subcommands) and the MySQL client (in terms of option-handling), so if you know how to use those tools, it's a cinch to learn. Basically you `skeema init` to initially populate your CREATE statements (one file per table) from a db instance. You can then change those files locally and run `skeema diff` to view auto-generated DDL, or `skeema push` to actually execute it. Or if you make "out of band" changes directly to the db -- such as renames, or just someone doing something manually for whatever reason -- then you can `skeema pull` to update the files accordingly.

Skeema's configuration supports online schema change tools, service discovery, sharding, various safety options, etc. I also hope to add integration with GitHub API at some point. That can provide a really nice "self service" model for schema management at scale.

Re: Better Database Migrations in Postgres

#56

Earlier quoted context omitted.

Alembic is a standalone tool written in Python. Having spent years on rails migrations and then using Alembic, I can say that alembic is really brilliant. It behaves like git - it has a branching and merge model for Migrations (in case multiple people work on it simultaneously).

Alembic is amazing, but AFAIK only works with its author's (world-beating) ORM, SQLAlchemy. I didn't think it was a general-purpose database change tool.

You could potentially use SQLA's ability to reflect your db to create the models either to get you started, or maybe in an automated fashion. Im sure there are loads of corner cases where it wouldn't work.

We're happy users of alembic. In general we use it to do the initial grunt work and then manually edit the migrations / split into phases ourselves. For some stuff that Alembic doesn't handle very well (like check constraints) we've extended Alembic to handle those automatically in the way we like.

Re: Better Database Migrations in Postgres

#57
post #25

I really just want a deterministic (or sync-based) migration tool. The only two I'm aware of are Innovartis DBGhost and RedGate SQL Compare (though RedGate requires a license everywhere it runs, whereas DBGhost only requires a license to compile the package). This stems from my work years ago with databases in on-premise products. Customers would modify the database schema, causing migration "up" scripts to fail, and…

Strongly agree that the sync-based approach to schema migrations is much better.

I presented on this exact topic last week at PostgresOpen 2017, and have written a schema diff tool for Postgres that supports this approach (https://github.com/djrobstep/migra).

I'd be curious to know if this is what you are getting at, and if it would solve your problem?

Re: Better Database Migrations in Postgres

#58

OT: Postgres is an incredible piece of open source software, but the official admin UI pgAdmin is in a state of complete chaos. Any suggestions for an OSS replacement?

Not an UI but I love pgcli (https://github.com/dbcli/pgcli), it's like psql but 10 times better. It works pretty well

Re: Better Database Migrations in Postgres

#59
post #25

I really just want a deterministic (or sync-based) migration tool. The only two I'm aware of are Innovartis DBGhost and RedGate SQL Compare (though RedGate requires a license everywhere it runs, whereas DBGhost only requires a license to compile the package). This stems from my work years ago with databases in on-premise products. Customers would modify the database schema, causing migration "up" scripts to fail, and…

> I've been looking for something similar for Postgres and MySQL/MariaDB without any luck, and it really surprises me there's not more interest in doing migrations this way.

skeema (suggested above by evanelias) and square/shift are two management tools for MySQL/MariaDB.

On the lower level, I'm authoring GitHub's gh-ost, an online schema migration tool, and we have Ruby on Rails + scripts automation around that. Described in https://www.percona.com/live/17/sessions/automating-schema-c... .

I suspect no matter what solution you'd use, you will always need to break migrations into parts that are not fully automated. e.g. when dropping a column you'd first deploy code that ignores said column, only then run the migration to actually drop the column.

Re: Better Database Migrations in Postgres

#60
Please correct me if I'm wrong, but aren't migrations a solved problem in databases that support triggers?

* create a new schema in a table.new

* install update triggers on table.old to write the same content into table.new

* backfill table.new from table.old

* swap table.new and table.old

Post reply on HN