Hacker Accessed AWS for $50k+ – AWS Ignoring Me

You are aws user for 3 years but don’t have aws rep?

In my case we’ve got a miner on our jenkin for a day. I just call my aws sales rep and he get me a free lunch and a few credit to pay the business support for 1 month, then open the ticket through that business support. At the end of the week aws gave us extra credits around 10% of our yearly usage.

I don’t think they will waived all yours 26k. Thats your dev team fault and also your finance team or whoever that don’t watch the billing. But they can give you a lot of aws credit for many reason (promising startup, loyal customer, big company, etc)

So, what are best practices to avoid this situation in the first place? MFA. Billing alerts when estimated charges are over expected spending amounts. Anything else? Seems like a small mistake here could really harm a small business. Are there good ways to detect access that hasn't yet been exploited? Someone mentioned monitoring API calls, but what I'd googled on that seems fairly broad.

I've been looking into it and I think it would be possible to do if the documentation were better. They have `Budget Actions`, but it's fairly new so the documentation and examples are lacking. Based on what I've learned for myself, I would say the starter rules are:

    - Create a root account that's only used for consolidated billing and account recovery.  Secure it with 2FA.  I use TOTP saved on my Yubikey and my backup Yubikey with the setting that requires a physical touch to generate a code.
    - Create organizational accounts for every day use.  The exact way to structure them can get complicated, but there's a fair bit of documentation on it.
    - Set up budget alerts and budget actions before you start using a resource type.
    - Only create users with permissions to access resource types with budget actions set up.

It's easy to say, but very hard to do in practice based on my experience. The biggest problem is that if you want to use a couple of services (ex: EC2, S3), you have the complexity of 1000 services, IAM policies, etc. jammed in your face right at the start and it's almost impossible to figure out what permissions you need to do something.

It reminds me of SELinux where the permissions are difficult enough to deal with that you can write an audit log while performing an action and simply enable all the permissions that were logged.

The second biggest problem is that runaway billing is far worse for small users than for large users and big tech only cares about other big users because that's where the money is. Everything revolves around catering to huge users who don't care if they need to hire a consultant to tame their AWS billing, so the smaller users and startups are left with systems that are far too complex to meet their needs.

I prefer the way Digital Ocean works, but there are some things you just can't do with them. For example, Lambdas and SES don't have good alternatives at DO.

I also like Cloudflare Workers since I find it significantly easier to reason about price in the context of cost per execution instead of the complex formula used for Lambdas, etc.. I think Cloudflare is in a very good position to claw market share from the big clouds, but their Workers Unbound is pretty much a copy of Lambdas, Functions, etc. in terms of pricing structure, so it looks like they might be starting to go after those fat egress charges that everyone else makes their money from.

create a subsidiary: YourCompany Web Services LLC with minimal capital (only used for AWS)

AWS is like a weapons cache you've stumbled upon in the middle of the desert, lots of fun, useful and interesting stuff in it but you're going to get yourself hurt if you don't take proper precautions.

This sounds like a cautionary tale. I have spending alarms on my personal account for this very reason, I'll know within 5-10 minutes if my monthly spend is going to break $50 because I've set up my alarms.

Your other option is to start a Cloudtrail and alarm on foreign IPs that are logging in, new IAM users and keys being created and changes to any alarms you have in place to check for this stuff. It won't necessarily stop it, but you'll be able to react a lot faster.