Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Adds Put-If-Match (Compare-and-Swap)

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 166 posts

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#51

Post by willglynn »

losteric wrote: post #36
jiggawatts wrote: post #32

Earlier quoted context omitted.

That will probably never happen because of the fundamental nature of blob storage. Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block. Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streamin…

Why does the architect of blob storage matter? The hash can be calculated as data streams in for the first write, before data gets dispersed into multiple physically stored blocks.

It is common to use multipart uploads for large objects, since this both increases throughput and decreases latency. Individual part uploads can happen in parallel and complete in any sequence. There's no architectural requirement that an entire object pass through a single system on either S3's side or on the client's side.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#52

Post by mdavidn »

JoshTriplett wrote: post #39
cmeacham98 wrote: post #33

Earlier quoted context omitted.

Is there any reason you can't enforce that restriction on your side? Or are you saying you want S3 to automatically set the name for you based on the hash?

> Is there any reason you can't enforce that restriction on your side? I'd like to set IAM permissions for a role, so that that role can add objects to the content-addressible store, but only if their name matches the hash of their content. > Or are you saying you want S3 to automatically set the name for you based on the hash? I'm happy to name the files myself, if I can get S3 to enforce that. But sure, if it were…

I think you can presign PutObject calls that validate a particular SHA-256 checksum. An API endpoint, e.g. in a Lambda, can effectively enforce this rule. It unfortunately won’t work on multipart uploads except on individual parts.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#53

Post by aphantastic »

revnode wrote: post #19
offmycloud wrote: post #13

If the default ETag algorithm for non-encrypted, non-multipart uploads in AWS is a plain MD5 hash, is this subject to failure for object data with MD5 collisions? I'm thinking of a situation in which an application assumes that different (possibly adversarial) user-provided data will always generate a different ETag.

MD5 hash collisions are unlikely to happen at random. The defect was that you can make it happen purposefully, making it useless for security.

Sure, but theoretically you could have a system where a distributed log of user generated content is built via this CAS//MD5 primitive. A malicious actor could craft the data such that entries are dropped.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#54

Post by stevefan1999 »

So...are we closer to getting to use S3 as a...you guessed it...a database? With CAS, we are probably able to get a basic level of atomicity, and S3 itself is pretty durable, now we have to deal with consistency and isolation...although S3 branded itself as "eventually consistent"...

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#55

Post by danielheath »

texthompson wrote: post #48
jiggawatts wrote: post #32

Earlier quoted context omitted.

That will probably never happen because of the fundamental nature of blob storage. Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block. Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streamin…

Why would you PUT an object, then download it again to a central server in the first place? If a service is accepting an upload of the bytes, it is already doing a pass over all the bytes anyway. It doesn't seem like a ton of overhead to calculate SHA256 in the 4092-byte chunks as the upload progresses. I suspect that sort of calculation would happen anyways.

S3 supports multipart uploads which don’t necessarily send all the parts to the same server.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#56

Post by mr_toad »

stevefan1999 wrote: post #54

So...are we closer to getting to use S3 as a...you guessed it...a database? With CAS, we are probably able to get a basic level of atomicity, and S3 itself is pretty durable, now we have to deal with consistency and isolation...although S3 branded itself as "eventually consistent"...

People who want all those features use something like Delta Lake on top of object storage.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#57

Post by willglynn »

texthompson wrote: post #48
jiggawatts wrote: post #32

Earlier quoted context omitted.

That will probably never happen because of the fundamental nature of blob storage. Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block. Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streamin…

Why would you PUT an object, then download it again to a central server in the first place? If a service is accepting an upload of the bytes, it is already doing a pass over all the bytes anyway. It doesn't seem like a ton of overhead to calculate SHA256 in the 4092-byte chunks as the upload progresses. I suspect that sort of calculation would happen anyways.

You're right, and in fact S3 does this with the `ETag:` header… in the simple case.

S3 also supports more complicated cases where the entire object may not be visible to any single component while it is being written, and in those cases, `ETag:` works differently.

> * Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-S3 or plaintext, have ETags that are an MD5 digest of their object data.

> * Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-C or SSE-KMS, have ETags that are not an MD5 digest of their object data.

> * If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption. If an object is larger than 16 MB, the AWS Management Console will upload or copy that object as a Multipart Upload, and therefore the ETag will not be an MD5 digest.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.h...

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#58

Post by texthompson »

danielheath wrote: post #55
texthompson wrote: post #48

Earlier quoted context omitted.

Why would you PUT an object, then download it again to a central server in the first place? If a service is accepting an upload of the bytes, it is already doing a pass over all the bytes anyway. It doesn't seem like a ton of overhead to calculate SHA256 in the 4092-byte chunks as the upload progresses. I suspect that sort of calculation would happen anyways.

S3 supports multipart uploads which don’t necessarily send all the parts to the same server.

Why does it matter where the bytes are stored at rest? Isn't everything you need for SHA-256 just the results of the SHA-256 algorithm on every 4096-byte block? I think you could just calculate that as the data is streamed in.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#59

Post by Nevermark »

poincaredisk wrote: post #37
CubsFan1060 wrote: post #28

Earlier quoted context omitted.

Thanks! Do they mention when the comparison is done? Is it before, after, or during an upload? (For instance, if I have a 4tb file in a multi part upload, would I only know it would fail as soon as the whole file is uploaded?)

I imagine, for it to make sense, that the comparison is done at the last possible moment, before atomically swapping the file contents.

I can imagine it might be useful to make this a choice for databases with high frequency small swaps and occasional large ones.

1) default, load-compare-&-swap for small fast load/swaps.

2) optional, compare-load-&-swap to allow a large load to pass its compare, and cut in front of all the fast small swap that would otherwise create an un-hittable moving target during its long loads for its own compare.

3) If the load itself was stable relative to the compare, then it could be pre-loaded and swapped into a holding location, followed by as many fast compare-&-swaps as needed to get it into the right location.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#60

Post by [deleted] »

offmycloud wrote: post #13

If the default ETag algorithm for non-encrypted, non-multipart uploads in AWS is a plain MD5 hash, is this subject to failure for object data with MD5 collisions? I'm thinking of a situation in which an application assumes that different (possibly adversarial) user-provided data will always generate a different ETag.

[deleted]
Post reply on HN