Libaws: A simpler way to declare AWS infrastructure

The hard part isn't making AWS easier, it is making a tool which is both easy and flexible enough that a user can do what they need (within reason), then outgrow your tool and move on. In specific any tool based abstraction has the downside that if the only way to interact with your infrastructure is through the tool then you severely limit the options for a user to make customizations. AWS will add features faster than you can keep up, and there will always be someone who can't use your tool unless you have feature X, Y, and Z.

At that point you can either add more and more features until your tool becomes as complex to use as all of AWS itself, or you can draw a line in the sand and say "you have outgrown my tool, it is time to move on". When they reach that point you should try to make it easy for them to avoid having to recreate everything from scratch themselves.

For AWS Copilot we tackled this problem in part by adding a "copilot svc package" command which spits out a full CloudFormation stack for your application and all its infrastructure, so that you don't have to restart from scratch if you decide you have outgrown Copilot. Copilot will help you deploy simple Docker based container applications quickly and easily, but then if you have more complex needs you can "graduate" on to the next experience by exporting things up into a stable, productionized CloudFormation template that lets you customize the full range of AWS settings and resources to your hearts content.

I think your tooling will need something similar. The "outgrowing" section (https://github.com/nathants/libaws#outgrowing) is a bit weak right now, and that's what you need to really nail down for people to have the trust to start adopting a tool for serious usage. They need to know that once they outgrow the tool they will have a viable path that won't be a huge headache.

Otherwise I like your ideas here, and I think the YAML based DSL here is a fascinating abstraction of many common, simple use cases. Great job!

The problem is more with the software that uses it, and it using YAML is just a high quality proxy for the signal you want.

YAML is a bit error prone, but it's those tools that have an incredibly complex format.

The classic example is public-read S3 buckets. My favorite-er example is actually a security guidance which reads: nothing should use the default VPC (usually phrased from the perspective of, all the SGs on the default VPC should restrict all traffic). Follow that line of logic: why did you give me a default VPC then? Well, they'd argue, it makes onboarding easier, you can just deploy all these cool resources without worrying about the complexities of the networking. Wait hold on. All of this is within their power to control, they made the networking insanely complex, they made the networking layer integral to the Cool Product, they created the idea of a default VPC, they created the security rule that says its bad, and it really paints this picture of a house of cards where no-one is in control, no one has vision, its just a bunch of heads arguing with each other about their worldview on how things should be.

Of course, one head yells "backwards compatibility, legacy, more knobs more knobs" and because its Amazon (and Microsoft) that head gets a megaphone.

Because of this, as the OP wishes, AWS accounts are an extraordinarily useless thing to hand over to a development team. They're nothing like a Heroku or DigitalOcean account; at best, they're so complex that you need a domain expert to understand what to do, so all that ends up living in the CloudOps Org. At worst, the team will shoot themselves in the foot, and maybe take the company's bank account (or worse, customer data) with them.

Thus, the New Hotness in BigBoyTech is to build a PaaS on top of AWS. Because Amazon isn't capable of doing it! Lightsail is a joke. Lambda is probably the closest they've gotten. This PaaS asserts the security, compliance, performance, architecture, etc opinions the company wants, and dev teams can build on it.

Fine, whatever, but at some point you have to question why we're so obsessed with inefficiency that we tolerate knobs we won't use, or we'll miss turning to Secure, when alternatives do exist. Do you know what a compliance official's wet dream is? "Hey man I need somewhere to store this file in the cloud" "Use Google Drive" DONE! They bought Google Drive. Maybe tweaked a couple settings. Secure by default. AWS is nothing like that, and it can take millions in engineering to get an active AWS account to that state (let alone delegated management of multiple accounts across an org).

It's actually collective insanity that we put up with it, when the only half-reason why it is so is because JP Morgan has a COBOL service built in 1987 that can only run if its hardwired with a telephone jack directly to an ISP interchange in Ashburn, and they're too cheap to modernize it so just tax the rest of the world by abusing Amazon's "never say No to a paying customer" policy. I'm being dramatic, but that describes 98% of the knobs on AWS.