Hacker Accessed AWS for $50k+ – AWS Ignoring Me • Hacker News Board

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#51

Post by mattmanser » Wed, May 12, 2021, 9:01 AM UTC

tonfreed wrote: post #45

thorin wrote: post #40

Earlier quoted context omitted.

You'll have a notice within 5-10 minutes if you continually carry your phone and are "supporting" your application 24/7. What if you want to go camping or turn your phone off when you go to bed or do a long drive or something?

No need to be facetious, friend. Were I a more paranoid man I'd hook a lambda into the SNS topic and terminate all ec2 instances, delete all S3 buckets, delete all IAM keys and regenerate and send me the root password. I'm not that worried though.

There's nothing facetious about his post, you've over-reacted to it.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#52

Post by mattmanser » Wed, May 12, 2021, 9:04 AM UTC

OmegaPG wrote: post #35

asdfthrowawayyy wrote: post #15

Earlier quoted context omitted.

Nonsense, AWS is an industry leader in use of the asterisk. With exception to perhaps S3 and AWS Lambda, the predominant majority of AWS services are cloaked with accounting legalese and hidden billing gotchas that can easily result in a several thousand dollar bill within days if you don't analyze every aspect of AWS service offerings including their EULAs and acceptable use policies. So yes, it was bad form for OP…

It is not in AWS interest to give too many options to their users to restrict the usage and expenditure. AWS can certainly do a better job in telling me how to optimise my AWS hardware and expenses In OPs case, if you search in Google, you will find this AWS hacking happen a lot with many users including me and AWS support was extremely kind to me to give additional credits to offset that expense.

If AWS hacking happens a lot it sounds like a problem for AWS to deal with by default, doesn't it? Perhaps even be forced to by law?

It would be trivial for AWS to force you to setup spending limits when setting up an account, one for an alert, one for a hard lock.

AWS billing is terrible too, even now I'm getting charged a few dollars on my personal account for who knows what, I've cancelled everything I can find. It's like whack-a-mole everytime you spin something up.

In the end that's basically theft by AWS but you can't make too much noise or your account gets cancelled.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#53

Post by usr1106 » Wed, May 12, 2021, 1:03 PM UTC

AWS is a bad service because they don't let the customer set billing limits.

It seems earning money from users' mistakes is part of their business model.

Yes, resource limits are not a silver bullet. Users will complain when their important service goes down because it would have gone slightly over budget during "normal" use. Implementing it in a reasonable way is not perfectly simple. Probably you want separate limits for network, storage, and processing as well as different ways to enforce the limit. Deleting all S3 data might not be what many users would want. They might still be willing to pay for keeping the existing data.

And obviously changing the limit must not be possible with the same credentials that allow you to use resources. Another fundamental challenge.

But with the size of AWS's business there is no excuse not to implement anything. They just value profit over customers.

(Sorry, not a reply to the original poster's question. I don't have anything significantly different from what has been mentioned by others.)

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#54

Post by usr1106 » Wed, May 12, 2021, 1:41 PM UTC

thorin wrote: post #40

tonfreed wrote: post #29

Earlier quoted context omitted.

AWS is like a weapons cache you've stumbled upon in the middle of the desert, lots of fun, useful and interesting stuff in it but you're going to get yourself hurt if you don't take proper precautions. This sounds like a cautionary tale. I have spending alarms on my personal account for this very reason, I'll know within 5-10 minutes if my monthly spend is going to break $50 because I've set up my alarms. Your other…

You'll have a notice within 5-10 minutes if you continually carry your phone and are "supporting" your application 24/7. What if you want to go camping or turn your phone off when you go to bed or do a long drive or something?

I don't want 5-10 minutes necessarily.

But what would be best practices for billing alarms? I have used them in the past when I used a bit more of AWS, but I don't think I ever got one (which is good).

But it happens to me that I miss emails that I would have liked to read in time for weeks. That could happen with a billing alarm, too.

Maybe you could forward it to SNS with SMS delivery. But as a matter of fact SMS is one of the few services (if not the only one) with a spending limit. If that is reached you silently won't get SMSes anymore, I have experienced that.

Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me

#55

Post by oerpli » Wed, May 12, 2021, 2:30 PM UTC

asdfthrowawayyy wrote: post #13

scrollaway wrote: post #3

AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for…

AWS support completely sucks unless you are paying 10K+ per month for the enterprise support tier. My average ticket response with paid AWS Developer is probably 72+ hours, and that's just for the initial triage what's up query. Unless you are either a seasoned Linux developer with many years of Linux internals experience, or you have an enterprise level account, all of the lower class AWS support rungs are largely m…

I work at a company with enterprise support and still found their contact persons & support completely useless.

We had a few training and introduction sessions with their people and it usually ended with us tuning out/joking about their useless slides and presentations (with "inspiring" Jeff Bezos quotes and brags about the number of packages ordered on Amazon.com that have 0 relevance for anything our company would like from AWS) in a private channel.