Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The AWS S3 Denial of Wallet Amplification Attack

blog.limbus-medtec.com

Post reply on HN
Print view View original

61–70 of 97 posts

Re: The AWS S3 Denial of Wallet Amplification Attack

#61

Post by slt2021 »

vermilingua wrote: post #59
slt2021 wrote: post #57

Earlier quoted context omitted.

this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws m…

Then subsequent requests that hit the cache shouldn’t be charged by that logic.

s3 is a complex system, you could be hitting a different node with subsequent requests where this cache entry does not exist yet.

if you think egress is expensive, well storing data in RAM for cache purposes is 1000000x more expensive

a lot of stuff could be happening. Main problem is AWS (i think) is charging for egress out of S3 system, but customers are looking at their ingress at client side and there is mismatch

Re: The AWS S3 Denial of Wallet Amplification Attack

#63

Post by adverbly »

Sounds like they were using the Range header on large files. I have made systems in the past using exactly this pattern(without the intentionally dropped requests).

I hope this doesn't result in any significant changes as I really liked using this pattern for sequential data processing of potentially large blobs.

Re: The AWS S3 Denial of Wallet Amplification Attack

#64

Post by Twirrim »

surfingdino wrote: post #53

AWS APIs need a cleanup. I am constantly running into issues not documented in the official doc, boto3 docs, or even on StackOverflow. It's not even funny when a whole day goes by on trying to figure out why I see nothing in the body of a 200 OK response when I request data which I know is there in the bowels of AWS. Then it turns out that one param doesn't allow values below a certain number, even though the docs sa…

Historically, they've been scared of versioning their APIs (not many services have done it, dynamodb has, for example).

It leads to a "bad customer experience", having to update lots of code, and also increases maintenance costs while you keep two separate code paths functional.

There's a lot about the S3 API that would be changed, including the response codes etc., if S3 engineers had freedom to change it! I remember many conversations on the topic when I worked alongside them in AWS.

Re: The AWS S3 Denial of Wallet Amplification Attack

#65

Post by waiwai933 »

andrewstuart wrote: post #17

Why is anyone using S3 when Cloudflare R2 is free?

R2 bandwidth is free, but storage is not.

R2 also doesn't have all the features that S3 does - including an equivalent of S3 Glacier, which is cheaper storage than R2. R2 also doesn't have object tagging, object-level permissions, or object locking. Sure, you could build your own layer in front of R2 that gives you these features, but are you necessarily saving money over just using S3?

Re: The AWS S3 Denial of Wallet Amplification Attack

#66

Post by andrewxdiamond »

Twirrim wrote: post #64
surfingdino wrote: post #53

AWS APIs need a cleanup. I am constantly running into issues not documented in the official doc, boto3 docs, or even on StackOverflow. It's not even funny when a whole day goes by on trying to figure out why I see nothing in the body of a 200 OK response when I request data which I know is there in the bowels of AWS. Then it turns out that one param doesn't allow values below a certain number, even though the docs sa…

Historically, they've been scared of versioning their APIs (not many services have done it, dynamodb has, for example). It leads to a "bad customer experience", having to update lots of code, and also increases maintenance costs while you keep two separate code paths functional. There's a lot about the S3 API that would be changed, including the response codes etc., if S3 engineers had freedom to change it! I remembe…

It’s quite insane the levels of effort S3 engineers put in to maintain perfect API compatibility. Even tiny details such as whitespace or ordering have messed up project timelines and blocked important launches.

Re: The AWS S3 Denial of Wallet Amplification Attack

#67

Post by [deleted] »

fabian2k wrote: post #40
paulddraper wrote: post #36

Earlier quoted context omitted.

Certainly if you are charging for internet egress . Like, charging for requests or internal data processing, sure okay. But this is a charge specifically for the data transferred from AWS to the internet . So if you're not transferring data to the internet....

The part where I think there is some flexibility is about the difference between "bytes attempted to transfer" and "bytes actually transferred". I think it is pretty fair to bill for the former, as long as you abort requests in a reasonable way. So I don't expect it to be billed exactly by the transferred byte, but I do expect it to not go above that higher than whatever the chunk size for transferring is.

[deleted]

Re: The AWS S3 Denial of Wallet Amplification Attack

#68

Post by paulddraper »

fabian2k wrote: post #40
paulddraper wrote: post #36

Earlier quoted context omitted.

Certainly if you are charging for internet egress . Like, charging for requests or internal data processing, sure okay. But this is a charge specifically for the data transferred from AWS to the internet . So if you're not transferring data to the internet....

The part where I think there is some flexibility is about the difference between "bytes attempted to transfer" and "bytes actually transferred". I think it is pretty fair to bill for the former, as long as you abort requests in a reasonable way. So I don't expect it to be billed exactly by the transferred byte, but I do expect it to not go above that higher than whatever the chunk size for transferring is.

Sure. In this case specifically AWS is attempting to transfer 70Gbps through a 1Gbps pipe. Not a rounding error.

Re: The AWS S3 Denial of Wallet Amplification Attack

#69

Post by paulddraper »

scottlamb wrote: post #19
yonixwm wrote: post #15

Earlier quoted context omitted.

So I guess the attack of the OP is a case AWS calculate price based on the unbounded request header and not on the actual egress

More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article…

> AWS customers are not billed for the data actually transferred to the Internet but instead for some amount of data that is cached internally.

But egress fees only apply to S3 transfers outside AWS?

So which is it? Data transferred to the Internet? Or data processed internally?

Re: The AWS S3 Denial of Wallet Amplification Attack

#70

Post by nijave »

kazen44 wrote: post #55
nijave wrote: post #37

Earlier quoted context omitted.

In all fairness, systems administrators have always had to pay for unauthorized requests and systems to mitigate the risk The new thing is hyperscalers have so much capacity you can get flooded by these long before the service degrades or goes offline

Also, the cost of doing this per request is insane compared to either absorbing or rate-limiting the bandwith the requests take. Cloud computing charges you by the request/byte/cpu cycle. Servers do not have this issue. Also, is it simply not possible to rate limit this on a per IP basis? Make client only able to do X requests per second from each unique IP/network flow.

>Cloud computing charges you by the request/byte/cpu cycle. Servers do not have this issue.

Sure they do. Processing requests takes bandwidth, CPU, memory, disk I/O

>Also, is it simply not possible to rate limit this on a per IP basis

It's largely useless. You'll block any legitimate bits/programs, people on CGNAT, people on corporate networks & bad actors will use botnets, residential IPs, VPNs to gain access to thousands or millions of unique IPs

Post reply on HN