81–90 of 124 posts
It astounds me that aws doesn't have a way to do this built in. Or even just a way to list all the resources you have, across all services in an account, without having to make thousands of API calls and hope you didn't miss a resource type.
99% of our AWS resources are terraformed, but developers constantly push back / want to use the console to create stuff to test or play around with. So we setup a separate "hack" AWS account and give them admin access in there, and have an automated job using AWS Nuke to delete everything in there once a quarter.
We give each one of our developers their very own aws account managed through AWS organizations service. They are full administrators and responsible for resources and cost. So far we haven’t had any issues or bad surprises, although we have setup some aws billing alerts just in case. Feel free to make them responsible for cost and resources and you’ll be surprised how well they can manage their own account.
I would love a `terraform --nuke`. Not `destroy` - the opposite - destroy/nuke what I don't have in my config.
Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using AWS nuke
so i have this weird monthly charge I cannot for the love of god find out what is causing it. I cancelled my credit card and now my credit score has taken a hit. All because some unknown AWS service that is in some zone that I cannot find at all and neither can AWS support.
Earlier quoted context omitted.
Sure, in an abstract sense maybe, but in a cross-service context it’s got nothing to do with their “cloud scale databases”. In fact, being able to delete a role without removing any associated resources is a feature, not a bug. And how would you even ensure referential integrity in this case - you would achieve the same effect by modifying the assume role policy but keeping the role around. You could craft a policy t…
Wedging a cloud resource so you can't delete it is always a bug.
Earlier quoted context omitted.
Yes, in the service that provides the AWS resource. Because it didn’t handle the fact that the role it’s using might be deleted or otherwise rendered un-assumable for a variety of different reasons at any point in time. Which is a feature. Not a bug.
I’m curious how you see this as a feature when it can get you into a very expensive and unresolvable situation; a AWS resource can’t be deleted and is running up costs. You’re at the mercy of AWS support.
I would love a `terraform --nuke`. Not `destroy` - the opposite - destroy/nuke what I don't have in my config.
That's one of my big complaints with terraform, and it's state-based system. It doesn't inform you if there are resources you don't know about.
terraform is best used on a per solution basis: one solution, one dedicated terraform project that will manage its state and its own state only. Multiple projects and people work carry out work in the same cloud account in parallel, and their terraform projects are not meant to interfere with each other. It works best for solutions that make use of fully managed cloud services.
Then there are also platform or connectivity level cloud resources (e.g. Transit Gateway and subnets that are mapped into the internal organisational network address space in AWS) that a random terraform project ought not to manage.
Lastly, if there is an actual need, a resource that terraform does not know about can be manually imported into the terraform's project state. This works best when infrastructure level resources were manually created a while ago and now have to be refactored into and managed by a terraform project. It is a tedious process that has to proceed with a lot of caution.
The caveat. It gets somewhat tricky when a non-serverless cloud resource requires an explicit subnet range allocation within an existing and managed CIDR or similar. There is no one solution to fit it all but containing such projects to their own dedicated VPC and setting up the VPC peering between a solution specific VPC and the main account VPC usually works satisfactory. That is, for example, how Kafka (AWS MSK) can be introduced into an AWS account without affecting existing CIDR mapping.
99% of our AWS resources are terraformed, but developers constantly push back / want to use the console to create stuff to test or play around with. So we setup a separate "hack" AWS account and give them admin access in there, and have an automated job using AWS Nuke to delete everything in there once a quarter.
Other accounts and environments (including dev) require everyone to follow a streamlined process: read only access to the account, a fully documented solution design and a corresponding terraform project in GitHub. terraform project checkin triggers a pull request for a review and an approval. Once the pull request has been scrutinised and merged, CI/CD runs terraform to provision resources in the account.