AWS Nuke – delete all resources associated with AWS account • Hacker News Board

Re: AWS Nuke – delete all resources associated with AWS account

#71

Post by deathanatos » Sun, Jul 03, 2022, 1:43 AM UTC

jiggawatts wrote: post #47

orf wrote: post #11

Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using AWS nuke

This is because neither AWS nor Azure use referential integrity in any of their "cloud scale" databases. For example, Azure uses some hideous JavaScript-based document DB where things like renames, moves, and deletes are hit & miss at best. A never-ending whack-a-mole of bugs and issues. Remember boys and girls: Being "cloud scale" means data corruption and referential integrity violation!

Not only referential integrity, but also not supporting Read-Your-Writes[1]. Cloud scale! Nothing like,

1. Create resource. Success.

2. Attempt to use/reference first resource in another resource/call: Failure: Referenced resource does not exist. Odd.

3. Create resource again? Failure! Resource already exists.

Our scripts have so many retry loops and arbitrary pauses mixed in to account for garbage like this. Distinguishing "did the call fail?" from "or is the system just lost in the land of eventual inconsistency?" ugh.

And yeah. shout out to Azure AAD where I can have a role assignment that is granting permissions to "unknown". We call them ghosts.

[1]: https://jepsen.io/consistency/models/read-your-writes

Re: AWS Nuke – delete all resources associated with AWS account

#72

Post by nouveaux » Sun, Jul 03, 2022, 2:09 AM UTC

101008 wrote: post #61

Frajedo wrote: post #38

Earlier quoted context omitted.

We give each one of our developers their very own aws account managed through AWS organizations service. They are full administrators and responsible for resources and cost. So far we haven’t had any issues or bad surprises, although we have setup some aws billing alerts just in case. Feel free to make them responsible for cost and resources and you’ll be surprised how well they can manage their own account.

> Feel free to make them responsible for cost and resources and you’ll be surprised how well they can manage their own account. Wow, this is horrible. I understand responsability but this is too much. Are other employees responsible if the company loses money for their actions?

It is quite common to have budgets employees have to work with.

Re: AWS Nuke – delete all resources associated with AWS account

#73

Post by OJFord » Sun, Jul 03, 2022, 2:21 AM UTC

iasay wrote: post #29

OJFord wrote: post #5

I would love a `terraform --nuke`. Not `destroy` - the opposite - destroy/nuke what I don't have in my config.

Oh that would be glorious. Not sure how it'd be possible though.

I've hitherto assumed it didn't because it wasn't, short of calling every single `*:Describe*` API anyway.

But the existence of aws-nuke makes me think (I haven't looked into what it's doing yet) there must be a better way of discovering used services/resources. Through billing perhaps?

Re: AWS Nuke – delete all resources associated with AWS account

#74

Post by zxcvbn4038 » Sun, Jul 03, 2022, 2:36 AM UTC

Marazan wrote: post #23

orf wrote: post #11

Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using AWS nuke

Even better if you have your cloud formation with the iam role in the same stack and there is something wrong the the rollback can happen such that the iam role gets deleted standing the compute environment and failing the rollback.

Update the stack to use a different role with cloudformation in its trust policy, then you can delete the stack with the new policy. Has to be two operations to work.

Re: AWS Nuke – delete all resources associated with AWS account

#75

Post by zxcvbn4038 » Sun, Jul 03, 2022, 2:45 AM UTC

orf wrote: post #28

zxcvbn4038 wrote: post #27

Earlier quoted context omitted.

Try logging into the console as root, that lets you delete anything. You can also try updating the cluster to have a new role.

That’s… not the issue. The cluster gets stuck in a deleting state forever. And at which point you’re unable to update it.

Time to open a support ticket, I’ve had a few things get into a bad state where they could not be updated or deleted, the backend team can step in, but first line support need to see the usual solutions fail before they escalate to them.

Re: AWS Nuke – delete all resources associated with AWS account

#76

Post by emersonrsantos » Sun, Jul 03, 2022, 2:52 AM UTC

Namespaces could solve all of that. Why AWS haven't added that yet? I understand that they rarely change the service, but c'mon, we're in 2022.

Re: AWS Nuke – delete all resources associated with AWS account

#77

Post by TedDoesntTalk » Sun, Jul 03, 2022, 2:57 AM UTC

orf wrote: post #70

XorNot wrote: post #69

Earlier quoted context omitted.

Wedging a cloud resource so you can't delete it is always a bug.

Yes, in the service that provides the AWS resource. Because it didn’t handle the fact that the role it’s using might be deleted or otherwise rendered un-assumable for a variety of different reasons at any point in time. Which is a feature. Not a bug.

I’m curious how you see this as a feature when it can get you into a very expensive and unresolvable situation; a AWS resource can’t be deleted and is running up costs. You’re at the mercy of AWS support.

Re: AWS Nuke – delete all resources associated with AWS account

#78

Post by TedDoesntTalk » Sun, Jul 03, 2022, 2:59 AM UTC

iasay wrote: post #19

orf wrote: post #11

Shout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using AWS nuke

There's loads of stuff like this. If you detach an EC2 instance from a virtual tape library and destroy it you can't delete the tape library any more. Even AWS support couldn't delete it. This is fine until you have 60TB of tapes online and are paying for it. Fortunately we found an EBS snapshot of the VM.

This is my worst nightmare. I had a situation like this where I was able to essentially ignore the entire AWS account and cancel the attached credit card. Amazon somehow linked this AWS account to my retail Amazon.com account and started booking my personal credit card.

Re: AWS Nuke – delete all resources associated with AWS account

#79

Post by thayne » Sun, Jul 03, 2022, 3:00 AM UTC

formerkrogemp wrote: post #67

efxhoy wrote: post #57

Earlier quoted context omitted.

Not even redshift enforces uniqueness on their primary keys or referential integrity on their foreign keys. That was a fun finding out…

> That was a fun finding out… Not my idea of a good time, but whatever floats your boat. Hopefully you had a backup?

That was almost certainly sarcasm.

Re: AWS Nuke – delete all resources associated with AWS account

#80

Post by thayne » Sun, Jul 03, 2022, 3:03 AM UTC

It astounds me that aws doesn't have a way to do this built in. Or even just a way to list all the resources you have, across all services in an account, without having to make thousands of API calls and hope you didn't miss a resource type.