Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The AWS S3 Denial of Wallet Amplification Attack

blog.limbus-medtec.com

Post reply on HN
Print view View original

91–97 of 97 posts

Re: The AWS S3 Denial of Wallet Amplification Attack

#91

Post by belter »

nielsole wrote: post #28
nicklecompte wrote: post #21

Earlier quoted context omitted.

This must be a regression bug in AWS's internal system. At a past job (2020) we used S3 to store a large amount of genomic data, and a web application read range requests to visualize tiny segments of the genetic sequence in relevant genes - like 5kb out of 50GB. If AWS had billed the cost of an entire genome/exome every time we did that, we would have noticed. I monitored costs pretty closely, S3 was never a problem…

S3 egress costs are free if the traffic stays within AWS. Sounds like your clients were EC2 instances so this wouldn't apply to you, would it?

> S3 egress costs are free if the traffic stays within AWS

Not if it's cross-region.

Re: The AWS S3 Denial of Wallet Amplification Attack

#92

Post by belter »

lulznews wrote: post #48

Does this apply to Cloudfront requests also?

Would seem to me that is also a Yes!...

CloudFront charges for all requests, so a 404 Not Found, will be counted towards the total number of requests, and you will be billed accordingly. Hopefully somebody will prove me wrong :-)

Re: The AWS S3 Denial of Wallet Amplification Attack

#93

Post by jpambrun »

martin-lim wrote: post #88
thayne wrote: post #84

Earlier quoted context omitted.

But the thing you are supposedly getting billed for is data transferred to the Internet. If the connection is closed there might be some data that goes out before the reset is received, but not that much. So either this is a bug, or the pricing documention is ... incorrect about what they actually charge you for.

Sadly this is no bug. Customers are NOT billed for actual data transferred. Customers are billed for "some kind of" data requested. If you interrupt the data transfer before downloading all requested data then it's "your fault". AWS documents this detail somewhere hidden on the S3 pricing page ( https://aws.amazon.com/s3/pricing/ ). Search for "Data Transfer Out may be different from the data received" in the "Data t…

It like if the gaz compagny billed you for more gaz your pipe can physically take. No amount of fine print can change physics. If they bill for some opaque internal metric they need to change the wording to reflect this on all documentation: "we will bill you for whatever we please, you can't audit and we won't explain how it works".

Re: The AWS S3 Denial of Wallet Amplification Attack

#94

Post by andersa »

thayne wrote: post #83
__roland__ wrote: post #27

Earlier quoted context omitted.

Sorry, I think that part of our write-up is misleading (I was involved in analyzing the issue described here). To our best understanding, what happens is the following: - A client sends range requests and cancels them quickly. - The full range request data will be billed (NOT the whole file), so I think this should read that the entire requested range gets billed, even if it never gets transferred (the explanation we…

Wait. What if the range is bigger than the file? Say if I made a request for a 5 TB range but the file is actually only 1kb. Would you get charged for 5 TB of egress?

This will return a range error instead of serving the file.

Re: The AWS S3 Denial of Wallet Amplification Attack

#95

Post by yencabulator »

slt2021 wrote: post #57
scottlamb wrote: post #19

Earlier quoted context omitted.

More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article…

this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws m…

If they charged for the extents that are actually read from S3 (even if the network connection was just closed), that'd be fine. But charging for terabytes when the actual connection consumed a few kilobytes is still wrong; surely they didn't e.g. buffer those terabytes of data from S3 disk storage into memory.

Charge for the buffer you fill: sure.

Charge the theoretical maximum: silly.

Re: The AWS S3 Denial of Wallet Amplification Attack

#96

Post by yencabulator »

akira2501 wrote: post #43
INTPenis wrote: post #26

Earlier quoted context omitted.

As it should be, but recently on HN it was posted that AWS will charge you for any unauthorized PUT request to your S3 buckets. Meaning even 4xx errors will rack up a charge. So your S3 bucket names must be hidden passphrases now that stand between an attacker and your budget.

Wow. Okay. New horrors brought to us by the modern world we've created. Thankfully, it does look like AWS is appropriately embarrassed over this, and is going to maybe do something. https://twitter.com/jeffbarr/status/1785386554372042890

Nah, they're just doing PR to manage their public image. AWS has known about that since 2006: https://twitter.com/cperciva/status/1785402732976992417

Re: The AWS S3 Denial of Wallet Amplification Attack

#97

Post by yencabulator »

nijave wrote: post #70
kazen44 wrote: post #55

Earlier quoted context omitted.

Also, the cost of doing this per request is insane compared to either absorbing or rate-limiting the bandwith the requests take. Cloud computing charges you by the request/byte/cpu cycle. Servers do not have this issue. Also, is it simply not possible to rate limit this on a per IP basis? Make client only able to do X requests per second from each unique IP/network flow.

>Cloud computing charges you by the request/byte/cpu cycle. Servers do not have this issue. Sure they do. Processing requests takes bandwidth, CPU, memory, disk I/O >Also, is it simply not possible to rate limit this on a per IP basis It's largely useless. You'll block any legitimate bits/programs, people on CGNAT, people on corporate networks & bad actors will use botnets, residential IPs, VPNs to gain access to tho…

The overhead of handling the 403 is very very small compared to what S3 charges for a PUT.
Post reply on HN