How an empty S3 bucket can make your AWS bill explode • Hacker News Board

Re: How an empty S3 bucket can make your AWS bill explode

#111

Post by tedmiston » Thu, May 02, 2024, 2:10 AM UTC

martin-lim wrote: post #81

tedmiston wrote: post #65

Earlier quoted context omitted.

What's crazier is that turning on Requester Pays does not actually mean requester pays when the request is a 403 [1][2]. Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill... That's a platform level security issue for AWS to resolve. We do not need another Bucketgate. [1]: https://twitter.com/Lauramaywendel/status/178507498639629151... [2]: https://docs.aws.…

For me it looks like AWS has mainly different scenarios to handle. First, unauthenticated users are a big security pain for AWS and S3 because S3 is used for everything. But S3 was not designed for "secure" public access. As a result billing attacks are possible (e.g. [1]) and will not be fixed (by design). Second, authenticated AWS users are kind of trustworthy and these are never assumed to do any DDoS stuff or sim…

> I am waiting for the moment when this assumption breaks because credit cards do not make users trustworthy.

IMO it is already broken, or beginning to crack at least, with the advent of throwaway virtual cards.

> Try to reduce the damage by activating Cost Anomaly Detection.

Yeah, this sounds like it could be a better solution than just using CloudWatch Billing alarms and billing alerts that I mentioned elsewhere. It's hard to tell at-a-glance if Cost Anomaly Detection is built on top of CloudWatch Billing alarms, or if they're just totally separate products. I don't seed the latter mentioned in the docs for the former.

[1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori....

Re: How an empty S3 bucket can make your AWS bill explode

#112

Post by martin-lim » Thu, May 02, 2024, 6:11 AM UTC

indigodaddy wrote: post #96

martin-lim wrote: post #81

Earlier quoted context omitted.

For me it looks like AWS has mainly different scenarios to handle. First, unauthenticated users are a big security pain for AWS and S3 because S3 is used for everything. But S3 was not designed for "secure" public access. As a result billing attacks are possible (e.g. [1]) and will not be fixed (by design). Second, authenticated AWS users are kind of trustworthy and these are never assumed to do any DDoS stuff or sim…

private in the sense that the connection will never make it to the S3 bucket, is that correct? private in the sense that it is locked down to allowed source IP's or IAM-- then the connection still gets there and AWS incurs a charge for the Access Denied response, etc

Yes, private in the sense of "block public access" (your second point) (https://docs.aws.amazon.com/AmazonS3/latest/userguide/access...)

Sadly, the private in the sense of "never make it to the S3 bucket" is nearly impossible. May this can be achieved by choosing some "secret" S3 bucket name.

Re: How an empty S3 bucket can make your AWS bill explode

#113

Post by Bombthecat » Thu, May 02, 2024, 8:09 AM UTC

tedmiston wrote: post #65

scottlamb wrote: post #55

Looking at AWS's price page for us-east-1 [1] * PUT, COPY, POST, LIST requests are $0.005 per 1,000 requests * GET, SELECT, and all other requests are $0.0004 per 1,000 requests It sounds like these are getting charged at the PUT request pricing. And IIUC, those prices are set because successful write operations seek on multiple HDDs (physically moving the head) to commit the data to multiple machines, or S3 would be…

What's crazier is that turning on Requester Pays does not actually mean requester pays when the request is a 403 [1][2]. Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill... That's a platform level security issue for AWS to resolve. We do not need another Bucketgate. [1]: https://twitter.com/Lauramaywendel/status/178507498639629151... [2]: https://docs.aws.…

That's why AWS doesn't allow pentesting of this (S3) service.

Re: How an empty S3 bucket can make your AWS bill explode

#114

Post by tedmiston » Thu, May 02, 2024, 7:09 PM UTC

EE84M3i wrote: post #104

tedmiston wrote: post #65

Earlier quoted context omitted.

What's crazier is that turning on Requester Pays does not actually mean requester pays when the request is a 403 [1][2]. Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill... That's a platform level security issue for AWS to resolve. We do not need another Bucketgate. [1]: https://twitter.com/Lauramaywendel/status/178507498639629151... [2]: https://docs.aws.…

What is bucketgate? Your comment comes up when searching...

I'm referring to the endless amounts of S3 buckets that ended up inadvertently exposing sensitive data publicly due to how complicated and confusing the various access control methods were / are. And also, Amazon very eventually doing something about, probably a decade after it was needed. [1][2][3]

[1]: https://cloudsecurityalliance.org/blog/2023/04/06/the-data-o...

[2]: https://news.ycombinator.com/item?id=18468753

[3]: https://news.ycombinator.com/item?id=34585670

Re: How an empty S3 bucket can make your AWS bill explode

#115

Post by nokya » Mon, May 06, 2024, 4:56 AM UTC

Lesson 4: mediocrity has become the most effective path to fame in a modern (and mediocre) world.

What's next? An article that explains why the bill stacks up when you log all denied fw packets?

Re: How an empty S3 bucket can make your AWS bill explode

#116

Post by busymom0 » Mon, May 06, 2024, 9:48 PM UTC

INTPenis wrote: post #77

tazu wrote: post #2

I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bucket names?

That's exactly it. I just spent the morning cleaning up my buckets. I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private. Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankf…

Reminder that if you are using pre-signed urls, then that url also exposes the bucket name. So, using hard to guess bucket names isn't a solution for many people.

Re: How an empty S3 bucket can make your AWS bill explode

#117

Post by busymom0 » Mon, May 06, 2024, 9:51 PM UTC

yenant wrote: post #89

Well this was an unwelcome attack vector, since I wrangle a fair few S3 buckets. I hastily threw together an S3 bucket name generator in Bash under Linux [1] to create names that follow the AWS rules to start my migration to my new S3 bucket names, and welcome any help to ensure it creates names that are strongly resistant to this attack, and follow various S3-service-like vendor rules, so I can do what I can until A…

Reminder that if you are using pre-signed urls, then that url also exposes the bucket name. So, using hard to guess bucket names isn't a solution for many people. And if you are using static websites, then amazon's own docs require you to use the website name as bucket name.

Re: How an empty S3 bucket can make your AWS bill explode

#118

Post by busymom0 » Mon, May 06, 2024, 9:57 PM UTC

fanf2 wrote: post #38

Use a password generator to create your S3 bucket names.

This doesn't solve the problem much.

1. If you are using buckets for static website hosting, amazon's own docs require you to use the website name as bucket name.

2. If you are using pre-signed urls, they expose the bucket name.

3. Bucket names must be between 3 (min) and 63 (max) characters long and have various limitations. So, it's not too hard to guess bucket names and run millions of requests without even having an account.