Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Atlas – Terraform but for Database Migrations

atlasgo.io

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Atlas – Terraform but for Database Migrations

#71

Post by daenney »

Though I understand where you're going with "Terraform for X", as someone who gets to deal with terraform on almost daily basis to manage infrastructure this makes me instantly want to head for the hills.

Terraform is certainly useful and when it works it works. The problem is that when it doesn't, things have a tendency to go horribly wrong. The amount of hours saved by terraform are barely offset by the amount of time spent recovering broken production infrastructure or resolving state issues because it threw in the towel halfway through some operation. That's without even digging into plan and apply discrepancies.

So, declarative schema migrations, yay, and market yourself that way. But I would be careful with being "terraform for X". There's a lot of terraform trauma among people who run production systems.

Re: Atlas – Terraform but for Database Migrations

#72

Post by LittlePeter »

I use migra [https://databaseci.com/docs/migra] for PostgreSQL.

I only put schema.sql into git, no migration scripts.

To make changes, I update schema.sql and load it up into temporary db. Then I run migra prod_db temp_db and it spits out the SQL statements that take me from the old schema.sql (which was in production) to the new. I eyeball the statements and if they look good, I apply them to prod_db.

Is Atlas able to handle foreign key definitions with ON UPDATE RESTRICT ON DELETE RESTRICT clause? Can it handle ID columns auto generate default always? Does it allow for constraints like check(airport_code = upper(airport_code))? The screenshots show rather simple SQL stuff.

How

Re: Atlas – Terraform but for Database Migrations

#73

Post by rotemtam »

daenney wrote: post #71

Though I understand where you're going with "Terraform for X", as someone who gets to deal with terraform on almost daily basis to manage infrastructure this makes me instantly want to head for the hills. Terraform is certainly useful and when it works it works. The problem is that when it doesn't, things have a tendency to go horribly wrong. The amount of hours saved by terraform are barely offset by the amount of t…

Hi daenney

(One of Atlas's creators here)

Thanks a lot for the feedback.

Frankly, we didn't make the Terraform analogy. We didn't even start this thread, just woke up to this wonderful discussion this morning :-) With regards to our marketing material, we only state that our DDL is terraform like because it is based on HCL..

I completely hear you on Terraform and it's pains and I've had my fair share of problems with it myself! This is why our approach to building Atlas is to offer two styles of migrations: declarative migrations (which is what you're seeing now) and a more advanced workflow (which I believe most mature projects will end up using) which we call migration authoring or versioned migrations.

This second approach is similar to existing migration tools, with versioned SQL files, up/down direction, etc. In fact, it will work great out of the box with most existing migration tools (such as Flyway, Liquibase etc.) The main difference is that users will be able to use Atlas's fairly advanced migration planner and more advanced features that will be available later this year.

So indeed, we're not a terraform for X, though i'm sure that title by whoever chose it brought this topic quite a bit of attention :-)

Re: Atlas – Terraform but for Database Migrations

#74

Post by Too »

jiggawatts wrote: post #26
ewuhic wrote: post #20

Earlier quoted context omitted.

Which identity management tools have you had in mind btw?

Over the years I've seen a lot of presentations from a lot of companies for all sorts of automation tools, especially identity management. Think "new user onboarding" for large enterprise. To be honest, I can't remember the names of any of the products. If it wasn't obvious from my rant, we didn't call any of them back and I promptly forgot about the specific vendors and their products. Most such products are utterly…

Luckily these days, this type of product is completely obsolete thanks to oauth and a central id provider.

Now the standard question to ask before buying any tool is. Do you support single sign on or do we need $complicatedprovisioningtool from your friends across the street? Ok. Bye.

Oauth isn’t perfect either, especially when it comes to propagating changes to a user. But it sure is a lot better than all these unique identity management tools doing even bigger mistakes over the rest api.

Re: Atlas – Terraform but for Database Migrations

#75

Post by majkinetor »

LittlePeter wrote: post #72

I use migra [ https://databaseci.com/docs/migra ] for PostgreSQL. I only put schema.sql into git, no migration scripts. To make changes, I update schema.sql and load it up into temporary db. Then I run migra prod_db temp_db and it spits out the SQL statements that take me from the old schema.sql (which was in production) to the new. I eyeball the statements and if they look good, I apply them to prod_db. Is Atlas abl…

You eyeball the statements? Thats 100% unreliable, I'll just wait for your sugar to go down while you do that.

Migration that is not automatic is not an option.

Re: Atlas – Terraform but for Database Migrations

#76

Post by majkinetor »

jiggawatts wrote: post #26
ewuhic wrote: post #20

Earlier quoted context omitted.

Which identity management tools have you had in mind btw?

Over the years I've seen a lot of presentations from a lot of companies for all sorts of automation tools, especially identity management. Think "new user onboarding" for large enterprise. To be honest, I can't remember the names of any of the products. If it wasn't obvious from my rant, we didn't call any of them back and I promptly forgot about the specific vendors and their products. Most such products are utterly…

Yeah, IAM software suck quite a lot. The bigger the company, the worst it is.

Re: Atlas – Terraform but for Database Migrations

#77

Post by majkinetor »

It takes few days for one to create migration tool that uses SQL files using shell only (for example PoweShell being a serious language is awesome for this).

I did that on number of projects and it works like a charm. I can create any type of ad hoc workflow developer needs. For example most of them want functions/types/sprocs recreated each time on each migration which is trivial to do in shell. Some of the devs had really wired but useful wishes that I was able to trivially easy do. Afterall, its just executing sql scripts 1 by 1, optionally with transaction started before that.

Nobody ever used down migrations so not sure why people still do that.

Shell is mandatory because I may want to do any number of things around migrations that require feedback from it (for example installing DLL files on Sql Server server or forcing backup just before migration).

Re: Atlas – Terraform but for Database Migrations

#78

Post by nijave »

paulgb wrote: post #34
sverhagen wrote: post #29

Earlier quoted context omitted.

That's why we try to set our databases to deletion_protection=true, and make it hard to delete them. But your point still largely stands!

Interestingly, that's the default for GCP but not AWS, even though both providers are developed by Hashicorp. I was pleasantly surprised by how difficult it was to (even intentionally) delete my GCP database when I was starting to use terraform.

Terraform usually use cloud provider defaults if they're applicable.

There's also 2 deletion protection mechanisms for RDS (and maybe GCP DBs). AWS has built in deletion protection which is an attribute on the resource and Terraform has lifecycle protection which is a meta attribute you can put on any resource type.

The former disables the delete API (returns an error) on the AWS side and the latter prevents Terraform from running the destroy event (which includes replaces)

Re: Atlas – Terraform but for Database Migrations

#79

Post by nijave »

hinkley wrote: post #32
redact207 wrote: post #5

I'll never be comfortable with any tool for that automatically generates schema changes, as I'm just never sure at what point it decides to delete parts of my prod db. All of my migrations are dumb DDL statements rolled up into a version. I know exactly what the final state is as it gets run and used when integration testing, staging etc. It's boring but pretty bulletproof. I can rename a table and be confident it'll…

It doesn't even have to delete anything. What happens when it gets 53% of the way through whatever it's doing at it errors out? Now wtf do I do?

>What happens when it gets 53% of the way through whatever it's doing at it errors out?

That tends to be the norm with Terraform, too :)

If it's like Terraform, you still need to know exactly what the tool is doing and how the services behave under the hood.

Re: Atlas – Terraform but for Database Migrations

#80

Post by nijave »

choward wrote: post #15

I wish people would explain why they created their project and what their pain points were with existing alternatives. All I can find is "Contrary to existing tools, Atlas intelligently plans schema migrations for you, based on your desired state." What's so bad about writing an explicit migrations using something like Flyway? I'm a fan of declarative configuration for the most part but it doesn't seem that beneficia…

I used to work on a monolithic Rails app and migrations would build up over time. We'd "collapse" migrations by doing a schema dump every couple weeks.

Without it, they'd accumulate thousands of migrations which would take quite a while locally and in CI to rebuild the database

It's also much easier to programmatically inspect. In ActiveRecord, the model attributes are dynamically determined from the DB (for better or worse) so you need to also take any migrations into account to know what fields a model has. From an inspection standpoint, it's nice to have the desired representation in code

Post reply on HN