Hacker Accessed AWS for $50k+ – AWS Ignoring Me

I'm trying to get help anywhere I can and a friend recommended I post this here.

My business has used AWS for around 3 years and our normal usage is $1k per month in EC2 and S3. In early March a hacker accessed our AWS account through my login via an IP address in Austria (I'm in Austin, TX). They spun up 3 large instances of EC2 which began charging us $1k-$2k per day.

In mid-April, while reviewing our books for the month of March, I saw a $26k charge from AWS. I thought it was a typo as $2.6k and asked the accountant. She stated that was the correct amount. I immediately got my dev team involved and we discovered the 3 instances to which we did not have any access to and stopped them immediately.

I opened a support case immediately which somehow got posted twice. Because the case was posted twice, the support team marked both cases as duplicates. I reopened one of the cases, it was resolved again as a duplicate. This has now happened several times.

I Googled around looking for a way to escalate this matter and found the following emails and cc'ed them on May 5th with an urgent plea via the original support case thread with another summary of the issue and links to my cases with my phone number to no avail. ams-csdm@amazon.com ams-opsmanager@amazon.com, ams-director@amazon.com, ams-vp@amazon.com

That email was ignored and I'm not sure where I can turn to next. I've tweeted about this and tagged AWS here - https://twitter.com/csakon/status/1391873413107617799?s=20

I'm not sure where to go next, can anyone give me any advice?

AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out?

Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for the bill. If it hadn't been that, it'd have been ransomware, you probably got lucky.

Now, to see if your situation can be improved: Put up some dollars and get business support. Make a clear and polite case, from the beginning. Ask for a refund but you don't have grounds to demand it; if they issue one, it's a gesture of good will. They probably will issue one if you haven't had to ask for that before, but it reflects badly on everybody that cryptominers weren't caught for two months.

And before you create that ticket, make some billing alerts so you can show AWS support that this won't happen again.

I don't think i'm leaving anything out. It was my account (which now has had password changes and MFA set up), but I don't understand how there weren't red flags on the Austrian IP address login and the sudden spike in usage. I realize (now) that CloudWatch exists, but not sure why this isn't standard.

I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking the first went through.

Once access was made, we were completely unaware of their existence until I saw the charges and asked our devs about it. They said they didn't have any knowledge or access to these new instances.

I appreciate the advice, will upgrade the support and try again.

AWS gives you access to a lot of footguns, and they expect you to implement proper security practices, access controls, monitoring, billing alerts, etc.

Why? Because they serve just about every use case and scenario. I've myself spun up $5k/day resources on accounts that did four magnitudes less / day before that. There are some limits by default which you can get raised, but they're not going to prevent a $50k bill - at best, they're here to prevent a $500k one.

Anyway I agree they could make it clearer that you have to do all this crap yourself but it makes for a poor sales pitch.

I agree that it should have been noticed earlier from a few vantage points. I'm managing over a dozen contractors which varies from dev to marketing to customer support, operations, sales, financials, investor relations, etc. Everything could be better, but it's a business and the squeaky wheel gets the oil. I never fathomed the squeaky wheel was going to be AWS usage charges.

I myself have never created an instance, I set up the account then gave access to the devs. I only log into the account to provide new access to devs that need it and none of them are full time.

Ultimately the responsibility lies with me, but I would disagree that my dev team is clueless. Rather they're working on development, not watching what the servers are doing on a daily basis so I think that's a bit unfair.