Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The AWS S3 Denial of Wallet Amplification Attack

blog.limbus-medtec.com

Post reply on HN
Print view View original

21–30 of 97 posts

Re: The AWS S3 Denial of Wallet Amplification Attack

#21

Post by nicklecompte »

wmf wrote: post #5

"With range requests, the client can request to retrieve a part of a file, but not the entire file. ... Due to the way AWS calculates egress costs the transfer of the entire file is billed." WTF if true.

This must be a regression bug in AWS's internal system. At a past job (2020) we used S3 to store a large amount of genomic data, and a web application read range requests to visualize tiny segments of the genetic sequence in relevant genes - like 5kb out of 50GB. If AWS had billed the cost of an entire genome/exome every time we did that, we would have noticed. I monitored costs pretty closely, S3 was never a problem compared to EC2.

It also seemed like the root cause was an interrupted range request (although I wasn't fully clear on that). Even so that seems like a recent regression. It took me ages to get that stupid app working, I interrupted a lot of range requests :)

Re: The AWS S3 Denial of Wallet Amplification Attack

#23

Post by easton »

scottlamb wrote: post #19
yonixwm wrote: post #15

Earlier quoted context omitted.

So I guess the attack of the OP is a case AWS calculate price based on the unbounded request header and not on the actual egress

More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article…

> this completely fails to explain the article's observed 3 TB sent vs 130 TB billed

I interpreted that to be that their code was doing this over and over again, so in total they retrieved 3TB over a set of requests. Still horrifying, but mildly more explainable.

Re: The AWS S3 Denial of Wallet Amplification Attack

#26

Post by INTPenis »

akira2501 wrote: post #22

We use CloudFront and we deny public users the ability to access S3 directly. You can even use Signed URLs with CloudFront if you like. I'm not sure I'd evere feel comfortable letting the public at large hit my S3 endpoints.

As it should be, but recently on HN it was posted that AWS will charge you for any unauthorized PUT request to your S3 buckets. Meaning even 4xx errors will rack up a charge.

So your S3 bucket names must be hidden passphrases now that stand between an attacker and your budget.

Re: The AWS S3 Denial of Wallet Amplification Attack

#27

Post by __roland__ »

fabian2k wrote: post #9
wmf wrote: post #5

"With range requests, the client can request to retrieve a part of a file, but not the entire file. ... Due to the way AWS calculates egress costs the transfer of the entire file is billed." WTF if true.

That sounds egregious enough that I have trouble believing this can be correct. My understanding is that AWS bills for egress for every service, parts of the file that aren't transferred are not part of this so can't be billed. There could certainly be S3-specific charges that affect cases like this, no idea. But if AWS bills the full egress traffic costs for a range request I'd consider that essentially fraud.

Sorry, I think that part of our write-up is misleading (I was involved in analyzing the issue described here). To our best understanding, what happens is the following:

- A client sends range requests and cancels them quickly.

- The full range request data will be billed (NOT the whole file), so I think this should read that the entire requested range gets billed, even if it never gets transferred (the explanation we received for this is that it's due to some internal buffering S3 is doing, and they do count this as egress).

In any case, if you send and cancel such requests quickly (which is easy enough, this was not even an adversarial situation, just a bug in some client API code) the egress cost is many times higher than your theoretical bandwidth (and about 80x higher than in the AWS documentation, hence the blogpost).

Re: The AWS S3 Denial of Wallet Amplification Attack

#28

Post by nielsole »

nicklecompte wrote: post #21
wmf wrote: post #5

"With range requests, the client can request to retrieve a part of a file, but not the entire file. ... Due to the way AWS calculates egress costs the transfer of the entire file is billed." WTF if true.

This must be a regression bug in AWS's internal system. At a past job (2020) we used S3 to store a large amount of genomic data, and a web application read range requests to visualize tiny segments of the genetic sequence in relevant genes - like 5kb out of 50GB. If AWS had billed the cost of an entire genome/exome every time we did that, we would have noticed. I monitored costs pretty closely, S3 was never a problem…

S3 egress costs are free if the traffic stays within AWS. Sounds like your clients were EC2 instances so this wouldn't apply to you, would it?

Re: The AWS S3 Denial of Wallet Amplification Attack

#29

Post by jsheard »

Havoc wrote: post #24

It's almost like the combination of public accessible + charged per use + big cloud refusing to allow hardcaps on spend is a terrible idea...

Azure is probably the most egregious example of this, AWS and GCP can at least claim they have architectural barriers to implementing a hard spending cap, but Azure already has one and arbitrarily only allows certain subscription types to use it. If you have a student account then you get a certain amount of credit each month and if you over-spend it then most services are automatically suspended until the next month, unless you explicitly opt-out of the spending limit and commit to paying the excess out of pocket. However if you have a standard account you're not allowed to set a spending limit for, uh... reasons.

https://learn.microsoft.com/en-us/azure/cost-management-bill...

Re: The AWS S3 Denial of Wallet Amplification Attack

#30

Post by __roland__ »

nicklecompte wrote: post #21
wmf wrote: post #5

"With range requests, the client can request to retrieve a part of a file, but not the entire file. ... Due to the way AWS calculates egress costs the transfer of the entire file is billed." WTF if true.

This must be a regression bug in AWS's internal system. At a past job (2020) we used S3 to store a large amount of genomic data, and a web application read range requests to visualize tiny segments of the genetic sequence in relevant genes - like 5kb out of 50GB. If AWS had billed the cost of an entire genome/exome every time we did that, we would have noticed. I monitored costs pretty closely, S3 was never a problem…

You are right, this is about canceling range requests and still getting billed, not about requesting ranges and getting billed for the complete file egress. Sorry; we'll make the post clearer.
Post reply on HN