Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The AWS S3 Denial of Wallet Amplification Attack

blog.limbus-medtec.com

Post reply on HN
Print view View original

71–80 of 97 posts

Re: The AWS S3 Denial of Wallet Amplification Attack

#71

Post by klabb3 »

fabian2k wrote: post #40
paulddraper wrote: post #36

Earlier quoted context omitted.

Certainly if you are charging for internet egress . Like, charging for requests or internal data processing, sure okay. But this is a charge specifically for the data transferred from AWS to the internet . So if you're not transferring data to the internet....

The part where I think there is some flexibility is about the difference between "bytes attempted to transfer" and "bytes actually transferred". I think it is pretty fair to bill for the former, as long as you abort requests in a reasonable way. So I don't expect it to be billed exactly by the transferred byte, but I do expect it to not go above that higher than whatever the chunk size for transferring is.

That’s an orthogonal issue. There’s no interpretation of “egress” that means “stuff we do internally before leaving aws data centers”. If the tcp conn is reset only a few MB would leave aws frontend servers. Instead, it appears they’ve been basing the number off the range in the request and/or whatever internal caching/loading they’re doing within S3, which again has nothing to do with egress.

I mean, we already know egress is short for egregious. It’s an incredibly bad look to be overestimating the “fuck you” part of the bill.

Re: The AWS S3 Denial of Wallet Amplification Attack

#72

Post by Twirrim »

andrewxdiamond wrote: post #66
Twirrim wrote: post #64

Earlier quoted context omitted.

Historically, they've been scared of versioning their APIs (not many services have done it, dynamodb has, for example). It leads to a "bad customer experience", having to update lots of code, and also increases maintenance costs while you keep two separate code paths functional. There's a lot about the S3 API that would be changed, including the response codes etc., if S3 engineers had freedom to change it! I remembe…

It’s quite insane the levels of effort S3 engineers put in to maintain perfect API compatibility. Even tiny details such as whitespace or ordering have messed up project timelines and blocked important launches.

All meeting some random arbitrary, maybe not even conscious, decision made by an early S3 engineer when they were implementing something.

Re: The AWS S3 Denial of Wallet Amplification Attack

#73

Post by anon373839 »

bakugo wrote: post #60
anon373839 wrote: post #49

Earlier quoted context omitted.

These AI accusations are becoming a tired trope. What, exactly, about the article gives you the impression that it was generated by an LLM?

If writers don't want people to think their content is AI generated, maybe they shouldn't put ugly AI generated images on top of everything they write.

Ah, so it’s the illustrations?

Re: The AWS S3 Denial of Wallet Amplification Attack

#74

Post by ignoreusernames »

Early Athena (managed prestodb by AWS) had a similar bug when measuring colunar file scans. If it touched the file, it considered the whole file instead of just the column chunks read. If I’m not mistaken, this was a bug on presto itself, but it was a simple patch that landed on upstream a long time before we did the tests. This was the first and only time we considered using a relatively early AWS product. It was so bad that our half assed self deployed version outperformed Athena by every metric that we cared about

Re: The AWS S3 Denial of Wallet Amplification Attack

#76

Post by [deleted] »

__roland__ wrote: post #46
voidwtf wrote: post #3

The way billing is calculated should be clearly labeled along with the pricing. Azure does this too, it's super unclear what metric they're using to determine what will be billed for requests. We're having to find out via trial and error. If we request 0-2GB on a 6GB file, but the client cancels after 400MB. Are we paying 2GB or 400MB or 6GB? Is there a billed difference between Range: 0-, no "Range" header, and Rang…

Sorry for not having this made clearer (we'll fix this part of the post): the gotcha is not that AWS does not honor range requests, it's that canceling those will still add the full range of bytes to your egress bill (and this can add up quickly) although no bytes (or much fewer) have been transferred.

[deleted]

Re: The AWS S3 Denial of Wallet Amplification Attack

#77

Post by Havoc »

jsheard wrote: post #29
Havoc wrote: post #24

It's almost like the combination of public accessible + charged per use + big cloud refusing to allow hardcaps on spend is a terrible idea...

Azure is probably the most egregious example of this, AWS and GCP can at least claim they have architectural barriers to implementing a hard spending cap, but Azure already has one and arbitrarily only allows certain subscription types to use it. If you have a student account then you get a certain amount of credit each month and if you over-spend it then most services are automatically suspended until the next month…

Not just the student accounts - some of the corporate accounts have same. Had one with a $50 limit at my previous employer.

The simple truth is it's good for their earnings report if they have unrestricted access to your wallet

Re: The AWS S3 Denial of Wallet Amplification Attack

#79

Post by jpambrun »

slt2021 wrote: post #57
scottlamb wrote: post #19

Earlier quoted context omitted.

More or less. The article quotes AWS as saying the following: > Amazon S3 attempts to stop the streaming of data, but it does not happen instantaneously. ...which doesn't really explain it. It shouldn't send more than a TCP window after the connection is closed, and TCP windows are at most 1 GiB [1], usually much less, so this completely fails to explain the article's observed 3 TB sent vs 130 TB billed. The article…

this can be explained that this may be not egress out of AWS, but egress out of S3 system itself. S3 is a block storage, so retrieving an object for such a high availability and high perf service means it tries to pull some X block of data and cache it before sending through the socket. That X block of data is out of internal S3 storage, just not sent through the bigger Internet egress subsystem. So technically aws m…

The consequence of your interpretation is that surprise billing is not even bounded by the physical limitations. This is absolutely not reasonable. I am the user with in the zip case. I knew this testing would incur a cost, but I could imagine it could be 20x my network speed and I have absolutely no control over what AWS claims to meter.

Re: The AWS S3 Denial of Wallet Amplification Attack

#80

Post by jpambrun »

slt2021 wrote: post #61
vermilingua wrote: post #59

Earlier quoted context omitted.

Then subsequent requests that hit the cache shouldn’t be charged by that logic.

s3 is a complex system, you could be hitting a different node with subsequent requests where this cache entry does not exist yet. if you think egress is expensive, well storing data in RAM for cache purposes is 1000000x more expensive a lot of stuff could be happening. Main problem is AWS (i think) is charging for egress out of S3 system, but customers are looking at their ingress at client side and there is mismatch

But you are not charged if it's not Internet egress, which makes your argument about compute/ram cost moot.
Post reply on HN