Being able to check in a SQL file to our repo to manage all of our infra sounds like a dream. Added myself to the early access.
Why would you prefer to do this with SQL rather than any of the existing frameworks that already exist?
Cloud Infrastructure as SQL
61–70 of 113 posts
Re: Cloud Infrastructure as SQL
#62Earlier quoted context omitted.
personally, I love terraform. I dont like statefiles though. Its annoying to have them in a vault system
I'm frankly really confused that Terraform is still so widely used for AWS IaC when CDK exists.
CDK still uses cloudformation under the hood right? Anything to do with CF gets a hard pass from me.
Re: Cloud Infrastructure as SQL
#63What problem does this solve, as opposed to a git repositories? To me, declarative infrastructure management is incredibly important in order to reason about a deployment. Even though under the hood, yes, everything is stateful, it’s something I want abstracted away, not as a primary way of interacting with my infrastructure. I guess what I’m asking is what the canonical use case / target user of this is.
I think most large companies have written similar tools to deal with various use cases. As part of a security team I often have the need to query for what instance has been assigned what IP, what team owns what AWS account, which security groups have port X open. You can do all of this using API queries but it's tedious, slow, and you can run the risk of hitting API rate limits. Most of this information is not easily…
Just in case you don’t see it somewhere else in this thread. Someone posted something that does this: https://github.com/cloudquery/cloudquery
Re: Cloud Infrastructure as SQL
#64Steampipe ( https://steampipe.io ) is an open source CLI to query cloud infrastructure using SQL (e.g. AWS, GitHub, Slack, k8s, etc). It also has a HCL language to define security benchmarks and controls (e.g. AWS CIS, etc). We are a Postgres FDW under the hood with Go-based plugins, so write would be possible, but we've chosen to focus on read only so far. Definitely interested to see how you approach create, update…
We are using steampipe to connect 30 AWS accounts via SQL. In a single call we can query all our services. Using metabase to visualise it all. I highly recommend you give it a go
Re: Cloud Infrastructure as SQL
#65I agree with rubiquity, without getting into a philosophical debate about programming paradigms, we have started an open source project InfraQL (https://infraql.io/) which is a framework for all cloud operations (query and provisioning) using SQL, we support SELECT, INSERT and DELETE with UPDATE, UPSERT, REPLACE coming, totally different architectural approach to the other solutions referenced in this thread, supports config/data supplied via Json or Jsonnet. If we look at cloud resources being defined by configuration, and configuration as (simply) data, then a SQL dialect makes absolute sense - as opposed to continuously inventing new and complex frameworks and DSLs.
Re: Cloud Infrastructure as SQL
#66I'm not sure if this is useful at any capacity. Sorry to be hard on that.
There is good and bad. You're giving up conventional text based version control (assuming the 'database' behind this is anything like a conventional RDBMS database.) You could pick up a lot of flexibility; multiple, high feature frontends using anything that can deal with a relational schema, for example. Automation could benefit from the (loosely) standards based API that is SQL.
I think it's worth exploring. I suspect that if it had been done this way on day one people would be suspect of any approach that looks like the hodgepodge of bespoke cloud APIs we wade through today.
I've been working with OCI lately. Oracle Cloud Infrastructure if that still needs to be called out. Ironic that the company that popularized RDBMS software didn't invent this. They've got a really nice, first class fully supported Terraform provider, but apparently it didn't occur to anyone at Oracle that they might use their raison d'être to solve the problem. They could have done both even; the Terraform provider could be a front end for SQL DQL/DML operations.
The iasql blog makes the rather good point that behind all the kooky Cloud APIs and their vast walls of documentation and third party abstractions everything you're doing is actually landing in a SQL database anyhow. What, exactly, is the value of all that stuff in the middle?
There isn't enough info on the iasql site to illuminate this, but I think one problem they'll have is with types. There are a lot of special types in the world of cloud provisioning and if they try to shoehorn all those types into varchar, number and uuid the result will not be pleasing. Obvious examples would be IP addresses or CIDR notation.
Re: Cloud Infrastructure as SQL
#67Earlier quoted context omitted.
As someone who has to do lots of compliance activities, this is a fantastic tool, thanks!
Great to hear :-) Please check out our open source mods (e.g. AWS Compliance, GitHub Sherlock, DigitalOcean Thrifty) - we'd love your feedback & help! https://hub.steampipe.io/mods
Ooh this looks really powefull, any plans to add hetzner and backblaze?
Re: Cloud Infrastructure as SQL
#68Earlier quoted context omitted.
Infrastructure is quantum state. AWS APIs lie to you. Status pages lie to you. If I had a nickel every time the solution to a failing HTTP call was "just send it again"... Use whatever tool you like: SQL, Rust, carrier-pigeon - but it wont be a panacea to solving cloud infrastructure.
Everything in the real world is quantum state but that doesn’t stop every SaaS application out there from using an RDBMS as their system of record. This is why we have reconciliation processes. Terraform state files are just an ad-hoc storage format without any of the features that SQL and RDBMS have had for decades (I don’t mean to pick on terraform so much, it’s just the one I’m most familiar with).
Unless it can also do state verification, planning and the three-way: desired state, expected state and actual state in one diff, it doesn't hold a candle to Terraform.
At the same time this SQL would be useful when used in combination with a classic database migration framework which essentially comes down to a bad version of terraform.
Inversely, you could simply use de Terraform CDK and an SQL adapter to write SQL into Terraform definitions. People who want to use SQL can use that, and in the background it's still terraform with all its benefits.
Re: Cloud Infrastructure as SQL
#69Earlier quoted context omitted.
Like someone else says, SELECTs make sense, INSERT/UPDATE/DELETE to manage infrastructure state, rather than using “proper” infrastructure as code, sounds like a path to hell to me.
If properly modelled so the modifying operations are consistent and logical, SQL for modification has the advantage over "proper code to update" has the advantage that in many cases the same query to find something is used with only minor alteration to modify it. Now the real magic, would be to make changes atomic!
TF has locks so that two people cannot modify infra at the same time.
There are also dependencies within the resources to consider, such as network creation before firewall rule before VM
Re: Cloud Infrastructure as SQL
#70I see a lot of criticisms for not wanting to use SQL to do writes and I think that is misguided. The current state of your infrastructure is absolutely state and SQL is a great language for working with state. While Terraform and all these other "declarative" infrastructure tools are better than what came before them, you're ultimately playing Relation Stitcher by needing to connect the various pieces together. There…
Infrastructure is quantum state. AWS APIs lie to you. Status pages lie to you. If I had a nickel every time the solution to a failing HTTP call was "just send it again"... Use whatever tool you like: SQL, Rust, carrier-pigeon - but it wont be a panacea to solving cloud infrastructure.
I'll bite.
What infrastructure management tool do _you_ prefer then?