AMS is not a support org. Can you share your support case numbers? I may be able to escalate this to the right people.
Hacker Accessed AWS for $50k+ – AWS Ignoring Me
11–20 of 55 posts
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#12Earlier quoted context omitted.
I mean, it’s hard to have sympathy. Your setup sounds like a complete mess especially if you didn’t notice it. You didn’t even have MFA and you’re using the console to create resources? It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t…
I agree that it should have been noticed earlier from a few vantage points. I'm managing over a dozen contractors which varies from dev to marketing to customer support, operations, sales, financials, investor relations, etc. Everything could be better, but it's a business and the squeaky wheel gets the oil. I never fathomed the squeaky wheel was going to be AWS usage charges. I myself have never created an instance,…
Let me get this right: you give a parade of short-term contractors access to your production AWS account, presumably without proper permission segmentation, and neglect to do anything _other_ than that?
I assume you revoke access later, but I doubt you audit anything that they may have done (like create keys that outlive their access) in the account or that any of it is version controlled or traceable.
And you’re surprised you’re in this situation?
And fair enough, you pay your contractors to do a specific job. None of them are going to point out that the way you’re managing your infrastructure is pretty slow and inefficient, or that perhaps there’s a better way to do any of what you’re doing on AWS that is cheaper, faster, more secure and that might give you a far quicker iteration time with the added advantage that you won’t fall apart with a surprise bill like this. They, after all, are working on “development”.
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#13AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for…
AWS support completely sucks unless you are paying 10K+ per month for the enterprise support tier.
My average ticket response with paid AWS Developer is probably 72+ hours, and that's just for the initial triage what's up query.
Unless you are either a seasoned Linux developer with many years of Linux internals experience, or you have an enterprise level account, all of the lower class AWS support rungs are largely meaningless.
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#14AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for…
AWS support completely sucks unless you are paying 10K+ per month for the enterprise support tier. My average ticket response with paid AWS Developer is probably 72+ hours, and that's just for the initial triage what's up query. Unless you are either a seasoned Linux developer with many years of Linux internals experience, or you have an enterprise level account, all of the lower class AWS support rungs are largely m…
Support being slow and support being bad are two different things.
Billing team is separate anyway. You get a better "treatment" if you're paying (as in, they will look at your case closer), but you have access to the same service regardless.
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#15Earlier quoted context omitted.
I don't think i'm leaving anything out. It was my account (which now has had password changes and MFA set up), but I don't understand how there weren't red flags on the Austrian IP address login and the sudden spike in usage. I realize (now) that CloudWatch exists, but not sure why this isn't standard. I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking t…
I mean, it’s hard to have sympathy. Your setup sounds like a complete mess especially if you didn’t notice it. You didn’t even have MFA and you’re using the console to create resources? It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t…
Nonsense, AWS is an industry leader in use of the asterisk. With exception to perhaps S3 and AWS Lambda, the predominant majority of AWS services are cloaked with accounting legalese and hidden billing gotchas that can easily result in a several thousand dollar bill within days if you don't analyze every aspect of AWS service offerings including their EULAs and acceptable use policies.
So yes, it was bad form for OP to not establish billing alerts, but you can't in any way say that AWS is transparent about most aspects of their service tiers.
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#16For the future create a CloudWatch billing alert so that this doesn't happen again.
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#17Earlier quoted context omitted.
I agree that it should have been noticed earlier from a few vantage points. I'm managing over a dozen contractors which varies from dev to marketing to customer support, operations, sales, financials, investor relations, etc. Everything could be better, but it's a business and the squeaky wheel gets the oil. I never fathomed the squeaky wheel was going to be AWS usage charges. I myself have never created an instance,…
Let me get this right: you give a parade of short-term contractors access to your production AWS account, presumably without proper permission segmentation, and neglect to do anything _other_ than that? I assume you revoke access later, but I doubt you audit anything that they may have done (like create keys that outlive their access) in the account or that any of it is version controlled or traceable. And you’re sur…
Economy of scale is an indispensable aspect of any competent engineering effort. Any reasonably adept AWS engineer would always include cost considerations for any change to AWS backend architecture, including helping the client decide which service tier makes the most sense for a particular application's use case...
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#18AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out? Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for…
I don't think i'm leaving anything out. It was my account (which now has had password changes and MFA set up), but I don't understand how there weren't red flags on the Austrian IP address login and the sudden spike in usage. I realize (now) that CloudWatch exists, but not sure why this isn't standard. I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking t…
> I realize (now) that CloudWatch exists, but not sure why this isn't standard.
CloudWatch is standard AWS, or what do you mean?
Amazon gives you all the tools to monitor your usage and management of the cloud. You did not use those tools, had sloppy security and now it is Amazon's fault?
Take it as an expensive lesson learned. And get someone who knows what they are doing in AWS to administer it. Maybe your company was pennywise pound foolish by not having someone who knew AWS?
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#19Earlier quoted context omitted.
I mean, it’s hard to have sympathy. Your setup sounds like a complete mess especially if you didn’t notice it. You didn’t even have MFA and you’re using the console to create resources? It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t…
Nonsense, AWS is an industry leader in use of the asterisk. With exception to perhaps S3 and AWS Lambda, the predominant majority of AWS services are cloaked with accounting legalese and hidden billing gotchas that can easily result in a several thousand dollar bill within days if you don't analyze every aspect of AWS service offerings including their EULAs and acceptable use policies. So yes, it was bad form for OP…
I mean, yes, AWS isn’t going to not take your money if you’re using their service. And no, it’s not some murky snake pit. There are some nuances but on the whole it’s pretty clear.
Anyway I was talking more about cloudwatch monitoring, including tracing all API calls. The billing is secondary: someone has been in his account and might have exfiltrated everything.
Re: Hacker Accessed AWS for $50k+ – AWS Ignoring Me
#20Forgive me for stating the obvious, but if a problem gets reported more than once, you don't close ALL reports as duplicates and ignore the problem.
Whether the original security breach was the user's fault or not, Amazon Support dropped the ball here.