This seems terrible. What about managing views, or stored procedures? AKA the actually complex stuff to manage migrations of.
Atlas – Terraform but for Database Migrations
61–70 of 92 posts
Re: Atlas – Terraform but for Database Migrations
#62This seems terrible. What about managing views, or stored procedures? AKA the actually complex stuff to manage migrations of.
Views are going to be supported in the SQL-DDL version (see my comments below). I don't have any info about stored-procedures/triggers/events atm.
Re: Atlas – Terraform but for Database Migrations
#63Migration tool capable of handling background migrations will be a miracle we all have been waiting for.
Example of background migrations is nicely explained in the gitlab handbook https://docs.gitlab.com/ee/development/post_deployment_migra...
Re: Atlas – Terraform but for Database Migrations
#64Earlier quoted context omitted.
> So while a tool like this might be good when you're just starting out, for a "real" app you want to avoid this sort of automation, because the tool is almost certainly not going to be smart enough. Tools can be smart enough -- it just requires a lot of work and domain expertise. For example, Facebook has used declarative schema management company-wide for over a decade, to manage schema changes for what is likely t…
Facebook can do that because they have a homogeneous environment using only MySQL. And they have a strict engineering discipline and strong DBA team to limit the allowed schema structure. Not so much for other companies. Disclaimer: I don't work for Facebook, but I used to work at Google and build its Cloud SQL service.
Maybe you can explain why there is still no in place upgrade for CloudSQL PostgreSQL?
Re: Atlas – Terraform but for Database Migrations
#65Earlier quoted context omitted.
I have unrelated request since you are planning to add sql parser to your project. Would it be possible to have sql parser as seperate library? I am in need of sql parser and so far i have only been able to get parsers for specific dialects like pingcap parser for mysql. I think sql parser that can support multiple different sql dialects would be a great addition to golang ecosystem.
I agree with that as well. The idea is to create an infrastructure for SQL parsers. Base parser will hold all standard structure and dialects can register custom clauses/statements. At the moment, I generate PEG files for each dialect, but this creates too much duplicate code, and does not allow sharing same types/objects between different dialects. I thought about keeping it on the same GitHub repository ( https://g…
+1, separate go module looks good to me.
Re: Atlas – Terraform but for Database Migrations
#66Earlier quoted context omitted.
Facebook can do that because they have a homogeneous environment using only MySQL. And they have a strict engineering discipline and strong DBA team to limit the allowed schema structure. Not so much for other companies. Disclaimer: I don't work for Facebook, but I used to work at Google and build its Cloud SQL service.
I probably should have disclosed in my previous comment that I'm a former member of Facebook's MySQL infra/automation team, although I didn't work on schema management there specifically. However, subsequent to FB, I independently built the declarative schema management tool skeema.io which is used by several large well-known companies. So Atlas is a competitor, and I may inherently be biased in my skepticism of tool…
You are right we are actually from the same clan which prefers pure SQL over another DSL.
FWIW, google does take the same approach as skeema to describe the desired schema state with pure SQL. But it's the Google Spanner does the heavy lifting. And that adds more respect for your work on skeema, since skeema is doing the heavy lifting from the tooling side, which in my opinion is more challenge.
BTW, I also need to disclose that I am currently building bytebase.com which is also a schema migration tool.
Re: Atlas – Terraform but for Database Migrations
#67Earlier quoted context omitted.
Facebook can do that because they have a homogeneous environment using only MySQL. And they have a strict engineering discipline and strong DBA team to limit the allowed schema structure. Not so much for other companies. Disclaimer: I don't work for Facebook, but I used to work at Google and build its Cloud SQL service.
Maybe you can explain why there is still no in place upgrade for CloudSQL PostgreSQL?
Actually I was the one leading to build their initial PostgreSQL offering. But I left the team later.
Technically, there's nothing prevent them doing that. It's probably just priority conflicts. A bit sigh though since it's a worthwhile feature IMO.
Re: Atlas – Terraform but for Database Migrations
#68Earlier quoted context omitted.
Hey, I'm one of the atlas's creator. Thanks for the feedback. I'm actually familiar with all the things you mentioned here (I worked at FB too ;)), and some of them are the reasons why we decided to create atlas and OSS it. First, atm, we support HCL and Go (with fluent API) for describing schemas, but in the next versions, we'll add support for SQL DDLs (e.g. "CREATE TABLE", "CREATE INDEX", etc). Can't promise time…
I have unrelated request since you are planning to add sql parser to your project. Would it be possible to have sql parser as seperate library? I am in need of sql parser and so far i have only been able to get parsers for specific dialects like pingcap parser for mysql. I think sql parser that can support multiple different sql dialects would be a great addition to golang ecosystem.
That would be really usefull! And when the parser does not understand a language construct (e.g. a new swl fwature) let it fall back to some dumb parsing for that part.
If it does not under „LIMIT 5 WITH TIES“ let it parse „LIMIT 5“ in it‘s usefull abstraction and just provide two suffix keywords „WITH“ and „TIES“.
Re: Atlas – Terraform but for Database Migrations
#69I'll never be comfortable with any tool for that automatically generates schema changes, as I'm just never sure at what point it decides to delete parts of my prod db. All of my migrations are dumb DDL statements rolled up into a version. I know exactly what the final state is as it gets run and used when integration testing, staging etc. It's boring but pretty bulletproof. I can rename a table and be confident it'll…
hey redact207
Thanks for the feedback!
(one of Atlas's creators here).
First of all, I completely understand you. Good ol' SQL has been with us for decades and isn't going anywhere. In the very near future you will be able to work with Atlas in pure SQL in a few ways: 1. Use a command like `atlas schema diff` (API not final) to generate the diff SQL for you, that you can then edit and use with your favorite tools. 2. Use a CREATE TABLE statement instead of HCL for your desired schema. 3. Use Atlas with a workflow that we call "migration authoring" to author for your the migrations into your directory format.
Second, I'd say that as much as I'd like it to be ubiquitous knowledge, many product teams don't have anyone on board with expertise like you probably have in planning migrations well, and so they can benefit greatly from a tool that will embed well tested and researched "DBA" knowledge into migration planning. The amount of outages I've heard about that are related to migrations that were ill planned is enormous.
Many have commented on this thread that it's a huge topic with a lot of subtleties and that it will be very hard to build a tool that does this well. I agree with that sentiment, but we built a strong team here at Ariga, and I hope together with the OSS community we will build something remarkable.
Re: Atlas – Terraform but for Database Migrations
#70I wish people would explain why they created their project and what their pain points were with existing alternatives. All I can find is "Contrary to existing tools, Atlas intelligently plans schema migrations for you, based on your desired state." What's so bad about writing an explicit migrations using something like Flyway? I'm a fan of declarative configuration for the most part but it doesn't seem that beneficia…
At my previous company there were several attempts to build declarative migration systems, one of which was more or less successful. They were all a lot more modest in scope and goals than Atlas.
The motivation is very straight forward: if you work in data management with different schemas, you often see similar migration patterns, so it makes sense to try to build abstractions representing these patterns.
If you couple this with the usual problem of the migration / schema duality, and some people who want to have a replica of the current state of the schema in the codebase in some way ("desired state"), you can see how we end up with schema-diff migration tools.
Whether these tools work well or not I think depends on the rate of schema changes vs size of the data store. The larger the dataset, the more specific you need to be with the migration strategies, so if you're not making schema changes very often, using a "clever", schema-diff migration tool adds risk for little benefit, and you might need to bypass the tool on a regular basis.
I don't work in a scenario in which such a tool would bring benefits, but there are projects that have frequent schema changes and data sizes small enough that you don't need to do anything special to maintain availability during migration.