Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The AWS S3 Denial of Wallet Amplification Attack

blog.limbus-medtec.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: The AWS S3 Denial of Wallet Amplification Attack

#81

Post by nextaccountic »

__roland__ wrote: post #30
nicklecompte wrote: post #21

Earlier quoted context omitted.

This must be a regression bug in AWS's internal system. At a past job (2020) we used S3 to store a large amount of genomic data, and a web application read range requests to visualize tiny segments of the genetic sequence in relevant genes - like 5kb out of 50GB. If AWS had billed the cost of an entire genome/exome every time we did that, we would have noticed. I monitored costs pretty closely, S3 was never a problem…

You are right, this is about canceling range requests and still getting billed, not about requesting ranges and getting billed for the complete file egress. Sorry; we'll make the post clearer.

This seems like something that customers can potentially sue for. AWS is overcharging them and not correcting the bug once they become aware of it.

Re: The AWS S3 Denial of Wallet Amplification Attack

#83

Post by thayne »

__roland__ wrote: post #27
fabian2k wrote: post #9

Earlier quoted context omitted.

That sounds egregious enough that I have trouble believing this can be correct. My understanding is that AWS bills for egress for every service, parts of the file that aren't transferred are not part of this so can't be billed. There could certainly be S3-specific charges that affect cases like this, no idea. But if AWS bills the full egress traffic costs for a range request I'd consider that essentially fraud.

Sorry, I think that part of our write-up is misleading (I was involved in analyzing the issue described here). To our best understanding, what happens is the following: - A client sends range requests and cancels them quickly. - The full range request data will be billed (NOT the whole file), so I think this should read that the entire requested range gets billed, even if it never gets transferred (the explanation we…

Wait. What if the range is bigger than the file? Say if I made a request for a 5 TB range but the file is actually only 1kb. Would you get charged for 5 TB of egress?

Re: The AWS S3 Denial of Wallet Amplification Attack

#84

Post by thayne »

nijave wrote: post #31
__roland__ wrote: post #27

Earlier quoted context omitted.

Sorry, I think that part of our write-up is misleading (I was involved in analyzing the issue described here). To our best understanding, what happens is the following: - A client sends range requests and cancels them quickly. - The full range request data will be billed (NOT the whole file), so I think this should read that the entire requested range gets billed, even if it never gets transferred (the explanation we…

This is a problem with lots of services. Blocking large quantities of legitimate looking requests is a hard problem. Request cancellation is also tricky and not supported well in a lot of frameworks/programming languages.

But the thing you are supposedly getting billed for is data transferred to the Internet. If the connection is closed there might be some data that goes out before the reset is received, but not that much. So either this is a bug, or the pricing documention is ... incorrect about what they actually charge you for.

Re: The AWS S3 Denial of Wallet Amplification Attack

#85

Post by scottlamb »

easton wrote: post #23
scottlamb wrote: post #19

Earlier quoted context omitted.

More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article…

> this completely fails to explain the article's observed 3 TB sent vs 130 TB billed I interpreted that to be that their code was doing this over and over again, so in total they retrieved 3TB over a set of requests. Still horrifying, but mildly more explainable.

Even so, TCP windows generally don't start at the maximum size, so I'm really not buying that ratio of bytes ACKed to sent.

One could disprove this explanation with a packet capture; just add the remaining window.

Re: The AWS S3 Denial of Wallet Amplification Attack

#86

Post by scottlamb »

paulddraper wrote: post #69
scottlamb wrote: post #19

Earlier quoted context omitted.

More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article…

> AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally. But egress fees only apply to S3 transfers outside AWS ? So which is it? Data transferred to the Internet? Or data processed internally?

Obviously they break the problem down into questions like "what's the maximum that could be transferred between any two components? what's the most expensive components this transits?" and pick the most expensive answers without any attempt at coherence.

Right now some class action lawyer is drooling.

Re: The AWS S3 Denial of Wallet Amplification Attack

#87

Post by [deleted] »

__roland__ wrote: post #27
fabian2k wrote: post #9

Earlier quoted context omitted.

That sounds egregious enough that I have trouble believing this can be correct. My understanding is that AWS bills for egress for every service, parts of the file that aren't transferred are not part of this so can't be billed. There could certainly be S3-specific charges that affect cases like this, no idea. But if AWS bills the full egress traffic costs for a range request I'd consider that essentially fraud.

Sorry, I think that part of our write-up is misleading (I was involved in analyzing the issue described here). To our best understanding, what happens is the following: - A client sends range requests and cancels them quickly. - The full range request data will be billed (NOT the whole file), so I think this should read that the entire requested range gets billed, even if it never gets transferred (the explanation we…

[deleted]

Re: The AWS S3 Denial of Wallet Amplification Attack

#88

Post by martin-lim »

thayne wrote: post #84
nijave wrote: post #31

Earlier quoted context omitted.

This is a problem with lots of services. Blocking large quantities of legitimate looking requests is a hard problem. Request cancellation is also tricky and not supported well in a lot of frameworks/programming languages.

But the thing you are supposedly getting billed for is data transferred to the Internet. If the connection is closed there might be some data that goes out before the reset is received, but not that much. So either this is a bug, or the pricing documention is ... incorrect about what they actually charge you for.

Sadly this is no bug. Customers are NOT billed for actual data transferred. Customers are billed for "some kind of" data requested. If you interrupt the data transfer before downloading all requested data then it's "your fault". AWS documents this detail somewhere hidden on the S3 pricing page (https://aws.amazon.com/s3/pricing/). Search for "Data Transfer Out may be different from the data received" in the "Data transfer" tab.

Re: The AWS S3 Denial of Wallet Amplification Attack

#89

Post by martin-lim »

jpambrun wrote: post #78

I had the exact same thing happen to me last Friday. My ISP metered 300GB, AWS measured 6,000GB. A factor of 20. I don't think this is reasonable. https://news.ycombinator.com/item?id=40203126#40205213 https://github.com/ZJONSSON/node-unzipper/issues/308

That could be the same root cause. You download data via range request but with no upper bound and AWS is billing you for very much more than you downloaded in reality.

Re: The AWS S3 Denial of Wallet Amplification Attack

#90

Post by thayne »

martin-lim wrote: post #88
thayne wrote: post #84

Earlier quoted context omitted.

But the thing you are supposedly getting billed for is data transferred to the Internet. If the connection is closed there might be some data that goes out before the reset is received, but not that much. So either this is a bug, or the pricing documention is ... incorrect about what they actually charge you for.

Sadly this is no bug. Customers are NOT billed for actual data transferred. Customers are billed for "some kind of" data requested. If you interrupt the data transfer before downloading all requested data then it's "your fault". AWS documents this detail somewhere hidden on the S3 pricing page ( https://aws.amazon.com/s3/pricing/ ). Search for "Data Transfer Out may be different from the data received" in the "Data t…

That explanation still makes it sound like you are charged for what is transferred to the Internet, but your application might not get some of it because it already closed the connection.

From the OP it sounds like the amount billed is too high to be explained by that.

If what you are billed for isn't actually data transferred, then it is deceptive to say that you are billed for "data transferred out".

Post reply on HN