Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How an empty S3 bucket can make your AWS bill explode

medium.com

Post reply on HN
Print view View original

81–90 of 118 posts

Re: How an empty S3 bucket can make your AWS bill explode

#81

Post by martin-lim »

tedmiston wrote: post #65
scottlamb wrote: post #55

Looking at AWS's price page for us-east-1 [1] * PUT, COPY, POST, LIST requests are $0.005 per 1,000 requests * GET, SELECT, and all other requests are $0.0004 per 1,000 requests It sounds like these are getting charged at the PUT request pricing. And IIUC, those prices are set because successful write operations seek on multiple HDDs (physically moving the head) to commit the data to multiple machines, or S3 would be…

What's crazier is that turning on Requester Pays does not actually mean requester pays when the request is a 403 [1][2]. Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill... That's a platform level security issue for AWS to resolve. We do not need another Bucketgate. [1]: https://twitter.com/Lauramaywendel/status/178507498639629151... [2]: https://docs.aws.…

For me it looks like AWS has mainly different scenarios to handle. First, unauthenticated users are a big security pain for AWS and S3 because S3 is used for everything. But S3 was not designed for "secure" public access. As a result billing attacks are possible (e.g. [1]) and will not be fixed (by design). Second, authenticated AWS users are kind of trustworthy and these are never assumed to do any DDoS stuff or similar. Here, we have even more surface for billing attacks. I am waiting for the moment when this assumption breaks because credit cards do not make users trustworthy.

My personal suggestion is that you should never use S3 for public access. Public buckets are an open gate to your AWS bill. If you have private buckets the attack surface is reduced but even in this situation you cannot prevent billing attacks. Try to reduce the damage by activating Cost Anomaly Detection.

[1] https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-a... [2] https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-...

Re: How an empty S3 bucket can make your AWS bill explode

#83

Post by immibis »

robertlagrant wrote: post #26
immibis wrote: post #22

Earlier quoted context omitted.

The right way is for unauthorized requests to not cost anything.

That makes no sense, though. Some network and compute are running for this to happen. Someone needs to pay the bill. It's not obvious it should be the cloud service, if that's not the agreement.

When thy receive a request to a non-existent bucket why don't they charge it to the lexicographically closest customer?

Re: How an empty S3 bucket can make your AWS bill explode

#85

Post by iampims »

robertlagrant wrote: post #26
immibis wrote: post #22

Earlier quoted context omitted.

The right way is for unauthorized requests to not cost anything.

That makes no sense, though. Some network and compute are running for this to happen. Someone needs to pay the bill. It's not obvious it should be the cloud service, if that's not the agreement.

AWS rakes in multi billion dollars in profit every year. They can definitely eat the costs of unauthorized requests to s3 buckets.

Re: How an empty S3 bucket can make your AWS bill explode

#86

Post by otterley »

efge wrote: post #84

Unfortunately you can't always keep your bucket name a secret, if you use presigned URLs to let your users directly download objects...

It’s a good idea to use a reverse proxy like Nginx to handle the transfer through an internal redirect, which will hide the bucket name and authentication credentials.

Re: How an empty S3 bucket can make your AWS bill explode

#87

Post by efge »

otterley wrote: post #86
efge wrote: post #84

Unfortunately you can't always keep your bucket name a secret, if you use presigned URLs to let your users directly download objects...

It’s a good idea to use a reverse proxy like Nginx to handle the transfer through an internal redirect, which will hide the bucket name and authentication credentials.

But this means data transiting your network instead of being directly fetched from AWS infrastructure, which for me mostly defeats the purpose of presigned S3 URLs.

Re: How an empty S3 bucket can make your AWS bill explode

#88

Post by otterley »

efge wrote: post #87
otterley wrote: post #86

Earlier quoted context omitted.

It’s a good idea to use a reverse proxy like Nginx to handle the transfer through an internal redirect, which will hide the bucket name and authentication credentials.

But this means data transiting your network instead of being directly fetched from AWS infrastructure, which for me mostly defeats the purpose of presigned S3 URLs.

If your server is on AWS, then the egress cost should be identical whether the transfer originates from S3 or EC2. But if your server is on premise and your host charges for egress (or has impactful bandwidth restrictions), I can understand where you're coming from, as you could end up paying twice.

Re: How an empty S3 bucket can make your AWS bill explode

#89

Post by yenant »

Well this was an unwelcome attack vector, since I wrangle a fair few S3 buckets. I hastily threw together an S3 bucket name generator in Bash under Linux [1] to create names that follow the AWS rules to start my migration to my new S3 bucket names, and welcome any help to ensure it creates names that are strongly resistant to this attack, and follow various S3-service-like vendor rules, so I can do what I can until AWS and others come out with a stronger defense.

[1] https://github.com/automaticit-anthonyyen1/s3-bucket-name-ge...

Re: How an empty S3 bucket can make your AWS bill explode

#90

Post by gsf »

belter wrote: post #39
tazu wrote: post #2

I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bucket names?

First of all, I agree this is all bananas:-) but it seems is the same with GCP and Azure. For Backblaze B2 it seems its unclear in the documentation. Willing to be corrected on this....

Not in GCP if https://cloud.google.com/storage/pricing is to be believed:

> Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.

Post reply on HN