Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacker Accessed AWS for $50k+ – AWS Ignoring Me

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#41

Post by TrueGeek »

icedchai wrote: post #39
Dah00n wrote: post #36

Earlier quoted context omitted.

How does this help?

If it happens again, you can stop a small problem before it becomes a very big one. Also, it is a good practice in general.

It's still crap though. If they spin up an instance billing $1k a day you'll find out after they've already billed several hundred, if not at least $1k. There needs to be a way to set actually limits, not alerts. You should be able to say "I'm not using this service, the limit should be $0."

Of course, if OP didn't have MFA / let their keys leak this may not have helped anyway if the hacker was able to just remove the limits.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#42

Post by i_have_an_idea »

atatatat wrote: post #33
ies7 wrote: post #21

You are aws user for 3 years but don’t have aws rep? In my case we’ve got a miner on our jenkin for a day. I just call my aws sales rep and he get me a free lunch and a few credit to pay the business support for 1 month, then open the ticket through that business support. At the end of the week aws gave us extra credits around 10% of our yearly usage. I don’t think they will waived all yours 26k. Thats your dev team…

> watch the billing As a busy founder, I'll just go with the providers that don't chain me to their dashboard/email instead of providing meaningful caps.

you can setup whatever caps and alerts you want, it's not hard

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#43

Post by icedchai »

TrueGeek wrote: post #41
icedchai wrote: post #39

Earlier quoted context omitted.

If it happens again, you can stop a small problem before it becomes a very big one. Also, it is a good practice in general.

It's still crap though. If they spin up an instance billing $1k a day you'll find out after they've already billed several hundred, if not at least $1k. There needs to be a way to set actually limits, not alerts. You should be able to say "I'm not using this service, the limit should be $0." Of course, if OP didn't have MFA / let their keys leak this may not have helped anyway if the hacker was able to just remove th…

If they had actual limits, you'd have people complaining about their sites getting shut down because someone broke into their account and spun up a bunch of instances. Or a developer did it accidentally. Alerts are much safer.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#44

Post by atmosx »

BugsJustFindMe wrote: post #26
scrollaway wrote: post #3

AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for…

> AWS support doesn't generally suck ... without good reason I don't know what "generally" means here or what you're basing the claim on, but I'm with an organization that pays a lot of money to AWS and they regularly ghost us after giving a wrong or incomplete "works for me"-style answer.

Same here. For 10k / month I would expect proper support and answers, instead we get access to an amazing "confidential newsletter" about 124344 projects we don't care about, problems we report are put on the 10-year plan and answers are usually links to documentation about the product per se.

I have no idea why we're paying.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#45

Post by tonfreed »

thorin wrote: post #40
tonfreed wrote: post #29

Earlier quoted context omitted.

AWS is like a weapons cache you've stumbled upon in the middle of the desert, lots of fun, useful and interesting stuff in it but you're going to get yourself hurt if you don't take proper precautions. This sounds like a cautionary tale. I have spending alarms on my personal account for this very reason, I'll know within 5-10 minutes if my monthly spend is going to break $50 because I've set up my alarms. Your other…

You'll have a notice within 5-10 minutes if you continually carry your phone and are "supporting" your application 24/7. What if you want to go camping or turn your phone off when you go to bed or do a long drive or something?

No need to be facetious, friend. Were I a more paranoid man I'd hook a lambda into the SNS topic and terminate all ec2 instances, delete all S3 buckets, delete all IAM keys and regenerate and send me the root password.

I'm not that worried though.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#46

Post by isbvhodnvemrwvn »

tonfreed wrote: post #29
csakon wrote: post #5

Earlier quoted context omitted.

I don't think i'm leaving anything out. It was my account (which now has had password changes and MFA set up), but I don't understand how there weren't red flags on the Austrian IP address login and the sudden spike in usage. I realize (now) that CloudWatch exists, but not sure why this isn't standard. I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking t…

AWS is like a weapons cache you've stumbled upon in the middle of the desert, lots of fun, useful and interesting stuff in it but you're going to get yourself hurt if you don't take proper precautions. This sounds like a cautionary tale. I have spending alarms on my personal account for this very reason, I'll know within 5-10 minutes if my monthly spend is going to break $50 because I've set up my alarms. Your other…

Are you going to know it within 5-10m? Billing updates can take hours to appear on my account (I noticed that with some new services I was using). Even CloudTrail can take longer than 5-10m.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#47

Post by isbvhodnvemrwvn »

bsder wrote: post #22
scrollaway wrote: post #3

AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for…

> They probably will issue one if you haven't had to ask for that before, but it reflects badly on everybody that cryptominers weren't caught for two months. As much as I'd like to agree with you, AWS makes controlling this WAY too stupidly difficult. Out here in the real world, many of us are part of startups that have 4 people and a dog. We wear many hats, and "AWS Billing Expert" is not one we have time for. I act…

Having billing alerts is the bare minimum I'd expect someone half-competent relying on pay-per-use resources to set up.

Hard stop on spending would delete all your data and would not cover things which are billed e.g. monthly. There are AWS budget actions which address some of the issues (e.g. can put a hard deny on any actions or stop your EC2 instances), admittedly a relatively new service.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#48

Post by isbvhodnvemrwvn »

caseymarquis wrote: post #23

So, what are best practices to avoid this situation in the first place? MFA. Billing alerts when estimated charges are over expected spending amounts. Anything else? Seems like a small mistake here could really harm a small business. Are there good ways to detect access that hasn't yet been exploited? Someone mentioned monitoring API calls, but what I'd googled on that seems fairly broad.

Monitoring API calls is typically reactive on AWS - e.g. GuardDuty relies on logs from CloudTrail, which are incomplete (e.g. sending a message to SQS is not logged at all) and log delivery is delayed as well. Nevertheless it should detect the specific issue described here fairly quickly, certainly in less than 2 months.

Overall the best defense is defense in depth - use MFA for all human accounts, use IAM roles wherever possible, don't put stuff in public subnets, use restrictive firewall rules, follow least privilege principle, use secrets manager or similar services for storing credentials. You could write a book about it. Many people pretty much have.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#49

Post by isbvhodnvemrwvn »

TrueGeek wrote: post #41
icedchai wrote: post #39

Earlier quoted context omitted.

If it happens again, you can stop a small problem before it becomes a very big one. Also, it is a good practice in general.

It's still crap though. If they spin up an instance billing $1k a day you'll find out after they've already billed several hundred, if not at least $1k. There needs to be a way to set actually limits, not alerts. You should be able to say "I'm not using this service, the limit should be $0." Of course, if OP didn't have MFA / let their keys leak this may not have helped anyway if the hacker was able to just remove th…

> You should be able to say "I'm not using this service, the limit should be $0."

In your organization, add a service control policy which denies access to services you don't use. This will prevent all member accounts from executing actions you don't want, including root users. You can also deny any action on any resource in region other than whatever you expect to use (with some exceptions due to legacy stuff).

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#50

Post by TrueGeek »

isbvhodnvemrwvn wrote: post #49
TrueGeek wrote: post #41

Earlier quoted context omitted.

It's still crap though. If they spin up an instance billing $1k a day you'll find out after they've already billed several hundred, if not at least $1k. There needs to be a way to set actually limits, not alerts. You should be able to say "I'm not using this service, the limit should be $0." Of course, if OP didn't have MFA / let their keys leak this may not have helped anyway if the hacker was able to just remove th…

> You should be able to say "I'm not using this service, the limit should be $0." In your organization, add a service control policy which denies access to services you don't use. This will prevent all member accounts from executing actions you don't want, including root users. You can also deny any action on any resource in region other than whatever you expect to use (with some exceptions due to legacy stuff).

this is great, thank you!
Post reply on HN