Noting that Azure Blob storage supports e-tag / optimistic controls as well (via If-Match conditions)[1], how does this differ? Or is it the same feature?
[1]: https://learn.microsoft.com/en-us/azure/storage/blobs/concur...
Amazon S3 Adds Put-If-Match (Compare-and-Swap)
81–90 of 166 posts
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#82Noting that Azure Blob storage supports e-tag / optimistic controls as well (via If-Match conditions)[1], how does this differ? Or is it the same feature? [1]: https://learn.microsoft.com/en-us/azure/storage/blobs/concur...
It's the same feature. Google Cloud Storage has it too: https://cloud.google.com/storage/docs/request-preconditions#...
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#83Earlier quoted context omitted.
Someone made an informed technical bet that worked out. Sounds like HN material to me. (Also, is it really a useful ad if you can't easily use the product?)
Worked out how? There’s no implementation. It’s just conjecture.
Pretty much all other S3 implementations (including open source ones) support this or equivalent primitives, so this is great for interoperability with existing implementations.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#84It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.
S3 has supported SHA-256 as a checksum algo since 2022. You can calculate the hash locally and then specify that hash in the PutObject call. S3 will calculate the hash and compare it with the hash in the PutObject call and reject the Put if they differ. The hash and algo are then stored in the object's metadata. You simply also use the SHA-256 hash as the key for the object. https://aws.amazon.com/blogs/aws/new-addit…
Unfortunately, for a multi-part upload it isn't a hash of the total object, it is a hash of the hashes for each part, which is a lot less useful. Especially if you don't know how the file was partititioned during upload.
And even if it was for the whole file, it isn't used for the ETag, so, so it can't be used for conditional PUTs.
I had a use case where this looked really promising, then I ran into the multipart upload limitations, and ended up using my own custom metadata for the sha256sum.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#85Earlier quoted context omitted.
> Is there any reason you can't enforce that restriction on your side? I'd like to set IAM permissions for a role, so that that role can add objects to the content-addressible store, but only if their name matches the hash of their content. > Or are you saying you want S3 to automatically set the name for you based on the hash? I'm happy to name the files myself, if I can get S3 to enforce that. But sure, if it were…
I think you can presign PutObject calls that validate a particular SHA-256 checksum. An API endpoint, e.g. in a Lambda, can effectively enforce this rule. It unfortunately won’t work on multipart uploads except on individual parts.
But in order to do that you need to already know the contents of the file.
I suppose you could have some API to request a signed url for a certain hash, but that starts getting complicated, especially if you need support for multi-part uploads, which you probably do.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#86What’s amazing is that it took them so long to add these functions
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#87Why is standard etag support making the frontpage?
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#88Now if only you had more control over the ETag, so you could use a sha256 of the total file (even for multi-part uploads), or a version counter, or a global counter from an external system, or a logical hash of the content as opposed to a hash of the bytes.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#89Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#90To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud Storage ~1 year ago. Our bet that S3 would get it in a reasonable time-frame worked out! https://turbopuffer.com/blog/turbopuffer
Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3
https://github.com/slatedb/slatedb will, I expect, use this at some point. Object backed DB, which is open source.