Live data from Hacker News

Cloud Infrastructure as SQL

iasql.com

101–110 of 113 posts

Re: Cloud Infrastructure as SQL

#101
post #54

Earlier quoted context omitted.

You're coming off a bit worked up, but I'll humor you anyway. I'm not going to type out a bunch of SQL as an example. SQL vs HCL isn't the point and they basically break even on expressiveness. After you've typed out your pseudo-JSON, what exactly are the existing tools saving you? From having to use some wrapper around the cloud API? That's the easy part. By overly focusing on SQL you're missing the forest for the t…

I think we're arguing two different points here. I'm not arguing that a theoretical better system exists that can plan infrastructure on a much more intelligent level than current public tooling does, or that you can't muck up terraform applies and cause downtime. I'm arguing that version controlling a bunch of SQL statements to describe your infrastructure feels wrong, outside of a few snappy short examples, and tha…

Why are Terraform modules convenient while stored procs are horrible though? They're doing the same job? E.g.

    call s3_bucket('bucket-name', true);
Is not really all that different from a Terraform module.

And the rest of your objections boil down to 'use in-code database migrations', which app developers have been doing as a best practice for a long time now.

Re: Cloud Infrastructure as SQL

#102

I see a lot of criticisms for not wanting to use SQL to do writes and I think that is misguided. The current state of your infrastructure is absolutely state and SQL is a great language for working with state. While Terraform and all these other "declarative" infrastructure tools are better than what came before them, you're ultimately playing Relation Stitcher by needing to connect the various pieces together. There…

SQL is great for querying (and steampipe exists just for that reason), but the point of Terraform is to be a desired state configuration system; SQL updates would remove its best feature. You could use the database as the desired state, but then it would no longer be readable text that you can version-control. Terraform code is equivalent to the desired content of the database, and running plan generates the required…

Database migrations are equivalent to the desired content of the database, and if I'm not mistaken, Terraform state files which are the _actual_ database, are not version controlled.

Re: Cloud Infrastructure as SQL

#103
post #80

Earlier quoted context omitted.

> yeesh INSERT/UPDATE/DELETEs would scare the heck out of me Out of curiosity, why would they?

1. Idempotency. Accidentally run an INSERT twice and you get two VMs instead of one. Sure you could be careful and do upsert-queries but is everyone in your org equally careful? 2. Reverts, in terraform it's as easy as removing the resource and re-applying. Or even just run teardown. Even if you've added other things in the meantime. With SQL if you've INSERTED something you need to come up with the corresponding DEL…

1. Use a unique key check that will fail if there's already a VM with the given key.

2. The OP mentioned that you can snapshot the database and restore it to a known-good state if anything goes wrong. State databases are probably not going to be humongous in size–you can imagine snapshotting them hourly for quite a long time.

3. How do you solve this problem today? By running Terraform applies locally on your machine to test out the modifications and then approving the PR?

4. That's exactly where foreign keys come in, they are exactly what enforces correct handling of dependencies between resources. E.g. some time ago one of our PagerDuty service monitors was renamed in Terraform. The plan destroyed the old monitor and created a new one with the new name, losing the linkage to the old incidents and making it look like the service had no incidents. Now imagine modelling this relationship in IaSQL and failing to destroy the old monitor because of the incidents.

Re: Cloud Infrastructure as SQL

#104
post #84

If I did infrastructure in SQL I’d accidentally leave a column out of a join and end up creating millions of dollars worth of machines I didn’t want.

Or, like some of today's tools it could ask you for confirmation of an execution plan before doing anything.

Re: Cloud Infrastructure as SQL

#105

isnt the big advantage to infrastructure as code the fact that you can version control it? and isnt it notoriously difficult to version control SQL? maybe I am missing something

It's actually really easy to version control SQL migrations, they are just plain text. It's what application developers who use databases have been doing for a long, long time. Check out Ruby on Rails migrations.

Re: Cloud Infrastructure as SQL

#106

Why not just a text file? Is this the main reason? > Unlike IaC, IaSQL makes the relations between pieces of your infrastructure first-class citizens, enforcing type safety on the data and changes to it. It seems easier to solve that problem via a source-control hook to check a text file, than move over to SQL. But maybe this proposal gets you other things too?

You get:

- CHECK constraints, e.g. you can make it impossible to misspell a VM type

- Type checks (depending on the database engine), e.g. you can't accidentally put a string where it wants an integer

- Foreign key checks so you can't delete some resource if others depend on it

- Atomic commits so the entire infra update happens in one shot or not at all.

Re: Cloud Infrastructure as SQL

#108
post #63
post #24

Earlier quoted context omitted.

I think most large companies have written similar tools to deal with various use cases. As part of a security team I often have the need to query for what instance has been assigned what IP, what team owns what AWS account, which security groups have port X open. You can do all of this using API queries but it's tedious, slow, and you can run the risk of hitting API rate limits. Most of this information is not easily…

Just in case you don’t see it somewhere else in this thread. Someone posted something that does this: https://github.com/cloudquery/cloudquery

Also https://steampipe.io/

Re: Cloud Infrastructure as SQL

#109
post #54

Earlier quoted context omitted.

I think we're arguing two different points here. I'm not arguing that a theoretical better system exists that can plan infrastructure on a much more intelligent level than current public tooling does, or that you can't muck up terraform applies and cause downtime. I'm arguing that version controlling a bunch of SQL statements to describe your infrastructure feels wrong, outside of a few snappy short examples, and tha…

Why are Terraform modules convenient while stored procs are horrible though? They're doing the same job? E.g. call s3_bucket('bucket-name', true); Is not really all that different from a Terraform module. And the rest of your objections boil down to 'use in-code database migrations', which app developers have been doing as a best practice for a long time now.

There are two decoupled parts to IAC - describing the state you want and reconciling that with the actual state of the world. Reconciling can be advanced as you want: you could take into account SLAs, perform progressive updates, or just ignore that and remove/create resources at will. So even if you used SQL and inserted it into a database, it's just a representation of state that needs to be reconciled in a separate step.

The problem is that "use in-code database migrations" is very very different to this problem. With database migrations you have a previous state and you manually code the steps to progress that state: add a column, then copy the data, then delete the old column, then do this, then do that. You do the reconciliation yourself.

And this is obvious when you think about it - imagine you did structure IAC like migrations, when you wanted to bring up some fresh infrastructure what happens? You would create all these resources, then modify them, then delete them, then create some new resources as a linear history of operations. Not good, very brittle.

So you need to decouple "what you want" from "how you get it". You could indeed use SQL to describe "what you want", but what does that really give you over using a language like Python/TypeScript/HCL to do it? It's not expressive enough and even things like conditionals are hard, and lets not even talk about string templating. Lets try and model the `s3_bucket` procedure with Postgres:

   CREATE PROCEDURE s3_bucket(bucket_name, bucket)
      LANGUAGE plpgsql AS
      $$
      DECLARE
         kms_arn string;
         bucket_arn string;
      BEGIN
        INSERT INTO aws_kms_key VALUES (nothing?) RETURNING arn INTO kms_arn;
        INSERT INTO aws_bucket (bucket_name, kms_key_id) VALUES ("bucket-name", kms_arn) RETURNING arn INTO bucket_arn;
        INSERT INTO aws_bucket_policy (bucket_arn, policy) VALUES (bucket_arn, make_policy_somehow?())
      END
      $$;
Of course, because you're describing a graph these returned arn's wouldn't actually _be_ arns, just placeholders that need to be reconciled later. And therein lies a problem - how do you ensure each node in your graph has a unique order-independent identifier? How do you know that you shouldn't need to create the `aws_kms_key` because it already exists, but you do need to create the `aws_bucket_policy` because you just added that `INSERT INTO` line? So you'd need to add a unique identifier to every `INSERT` statement:

    INSERT INTO aws_kms_key (node_id) VALUES ("this has to be unique else everything goes to hell") RETURNING arn INTO kms_arn;
Ugly. What's better about this than:

    resource "aws_kms_key" key {}

    resource "aws_s3_bucket" bucket {
        name = "bucket-name"
        kms_key_id = aws_kms_key.key.arn
    }
    
    resource "aws_bucket_policy" policy {
        bucket_id = aws_s3_bucket.bucket.id
        policy = jsonencode({...})  # or use the IAM document resource
    }
You could also model it with Pulumi:

    const key = new aws.kms.Key()
    const bucket = new aws.s3.Bucket("my-bucket");
    const policy = new aws.s3.BucketPolicy(bucket, make_policy())
Both of these are _much_ easier to grok and more flexible than the stored-procedure soup you will immediately get into if you use SQL.

And at the end of the day... it doesn't matter. You're not even _using_ a database to store your state, you're just using SQL to produce some graph of things you need to reconcile. You could write some huge stored procedure to turn your database state into a terraform/pulumi plan and get it to apply it with no problems.

Given that... what's the point of using SQL when there are better alternatives? Or, if you really hate yourself, go write it in COBOL. Equally as suitable.

Re: Cloud Infrastructure as SQL

#110
post #109

Earlier quoted context omitted.

Why are Terraform modules convenient while stored procs are horrible though? They're doing the same job? E.g. call s3_bucket('bucket-name', true); Is not really all that different from a Terraform module. And the rest of your objections boil down to 'use in-code database migrations', which app developers have been doing as a best practice for a long time now.

There are two decoupled parts to IAC - describing the state you want and reconciling that with the actual state of the world. Reconciling can be advanced as you want: you could take into account SLAs, perform progressive updates, or just ignore that and remove/create resources at will. So even if you used SQL and inserted it into a database, it's just a representation of state that needs to be reconciled in a separat…

> You're not even _using_ a database to store your state, you're just using SQL to produce some graph of things you need to reconcile.

Well, that's one way of doing it. The better way, imho, is to actually have a database to store the state. That also solves:

> You would create all these resources, then modify them, then delete them, then create some new resources as a linear history of operations. Not good, very brittle.

When you have an actual database backing your SQL queries, the migrations would all run in this DB, produce a final state of the DB, then this final state would get applied to bring up your system in one shot.

> So you'd need to add a unique identifier to every `INSERT` statement:

Sure, and that's fairly easy to derive, because we already have a unique identifier, the bucket name:

    INSERT INTO aws_kms_key (node_id) VALUES (concat('aws_kms_key', bucket_name)) RETURNING arn INTO kms_arn;
Post reply on HN