Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Adds Put-If-Match (Compare-and-Swap)

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 166 posts

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#31

Post by msoad »

rrr_oh_man wrote: post #30

Could anybody explain for the uninitiated?

It ensures that when you try to upload (or “put”) a new version of a file, the operation only succeeds if the file on the server still has the exact version (ETag) you specify. If someone else has updated the file in the meantime, your upload is blocked to prevent overwriting their changes.

This is especially useful in scenarios where multiple users or processes are working on the same data, as it helps maintain consistency and avoids accidental overwrites.

This is using the same mechanism as HTTP's `If-None-Match` header so it's easier to implement/learn

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#32

Post by jiggawatts »

JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

That will probably never happen because of the fundamental nature of blob storage.

Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block.

Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streaming the bytes to a central server to hash, the hash state is forwarded from block server to block server in sequence. Still though, it would be a very slow serial operation that could be fairly chatty too if there are many tiny blocks.

What could work would be to use a Merkle tree hash construction where some of subdivision boundaries match the block sizes.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#33

Post by cmeacham98 »

JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

Is there any reason you can't enforce that restriction on your side? Or are you saying you want S3 to automatically set the name for you based on the hash?

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#36

Post by losteric »

jiggawatts wrote: post #32
JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

That will probably never happen because of the fundamental nature of blob storage. Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block. Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streamin…

Why does the architect of blob storage matter? The hash can be calculated as data streams in for the first write, before data gets dispersed into multiple physically stored blocks.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#37

Post by poincaredisk »

CubsFan1060 wrote: post #28
Sirupsen wrote: post #27

Earlier quoted context omitted.

The short of it is that building a database on top of object storage has generally required a complicated, distributed system for consensus/metadata. CAS makes it possible to build these big data systems without any other dependencies. This is a win for simplicity and reliability.

Thanks! Do they mention when the comparison is done? Is it before, after, or during an upload? (For instance, if I have a 4tb file in a multi part upload, would I only know it would fail as soon as the whole file is uploaded?)

I imagine, for it to make sense, that the comparison is done at the last possible moment, before atomically swapping the file contents.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#39

Post by JoshTriplett »

cmeacham98 wrote: post #33
JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

Is there any reason you can't enforce that restriction on your side? Or are you saying you want S3 to automatically set the name for you based on the hash?

> Is there any reason you can't enforce that restriction on your side?

I'd like to set IAM permissions for a role, so that that role can add objects to the content-addressible store, but only if their name matches the hash of their content.

> Or are you saying you want S3 to automatically set the name for you based on the hash?

I'm happy to name the files myself, if I can get S3 to enforce that. But sure, if it were easier, I'd be thrilled to have S3 name the files by hash, and/or support retrieving files by hash.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#40

Post by fragmede »

mannyv wrote: post #38
sillysaurusx wrote: post #7

Finally. GCP has had this for a long time. Years ago I was surprised S3 didn’t.

GCP still doesn't have triggers out of beta last time i checked (which was a while ago).

Gmail was in beta for five years, I don't think that label really means anything.
Post reply on HN