Libaws: A simpler way to declare AWS infrastructure

A lot of effort is put into the customer experience, however often it's by the feature devs. So it's good once you grok it. But obtuse to start and not good across products.

It really needs some love from a higher level thinker who's not steeped in the solution, but in the problem. Presumably this is a PM but there's just not a lot of good ones in the org to be frank. They're almost not worth consulting.

Saying AWS should be easy is like saying a Lego parts bin should be easy. Easy for what? Easy to build a robot monkey with? Even if legos are really easy to snap together, you still have to do the work of putting them together in the shape of a robot monkey.

This project seems to be a collection of pre-assembled lego parts, that can eventually be put together in the form of a robot monkey. But I think all of us actually just want the robot monkey, not several clumps of legos. We should have more feature-complete projects that we can push a button and it pops up for you. But that's very different than saying AWS should be easy.

I think we should stop obsessing over "code", and start focusing on robot monkeys. In other words, stop thinking you need more and more and more abstractions around calling AWS APIs, and focus more on publishing a complete working solution for a specific problem.

I don't give a shit about SQS queues and CloudWatch alerts and Lambdas and IAM role policies! I give a shit about a file-upload-website that automatically triggers a python program to process a file and deliver the result into an S3 bucket or Postgres database. That complete thing is the robot monkey. That's what we should be publishing; not a million lines of boilerplate that I still have to spend time and energy cobbling together into a robot monkey. I have built the same robot monkey over and over and over again in 5 different abstractions. I don't care what the abstraction is, I just want the robot monkey!

I think the reason we don't have more robot monkeys is that they're built within companies out of "code", and so we aren't allowed to open-source them, because it's "intellectual property". We need a new abstraction that is not code, and is not considered super secret proprietary information, so we can publish it as open source without thinking.

For example: what if you could point a tool at some part of your AWS account, and it exports a declarative configuration file that is a snapshot of how everything is plumbed together? The tool could strip out any "literals" and anonymize all strings, so that there is nothing proprietary in the end result except a lot of API calls to functions with random names. The end result would essentially be an unlabeled architectural diagram that you could execute. You could then publish that on the Robot Monkey Database with a description of what it does.

You'll always be beholden to egress costs

...and doing that allows one to interface to existing sources of truth for inventory and whatnot.

A lot of Go code using the Go AWS SDK that's useful as a non-trivial example in and of itself.

Is this IaC "light" for purposes of setting up and tearing down small applications for experiments or demos?

Obviously a lot of effort spent. Of course, once you know AWS well enough to "make it easy", by that time, you've found the need for tooling which supports the harder stuff. And, frankly, someone new to AWS needs a lot of context just to understand the canonical ways to wire stuff together in AWS to even comprehend which pattern will be useful.

For example, S3 events are really backed by SQS under the covers, with implied retries on failure, but no DLQ support, and a lambda that triggers on an arriving S3 event may need to account for "duplicate" events, etc. That's just one wrinkle that even a simplified configuration might require more complex understanding of the underlying AWS ecosystem.